2024-05-27T06:30:09.997 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-27T06:30:09.998 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620 --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --name vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi -- /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/orig.config.yaml 2024-05-27T06:30:10.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T06:30:10.153 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620 branch: wip-vshankar-testing-20240521.063558-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: vshankar@redhat.com first_in_suite: false job_id: '7727620' kernel: flavor: default kdb: 1 sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20240521.063558-debug ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - Replacing daemon mds sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240521.063558-debug sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 owner: scheduled_vshankar@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 2499 sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 sleep_before_teardown: 0 subset: 573/1024 suite: fs suite_branch: wip-vshankar-testing-20240521.063558-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3zEpWVdrcS2uY7H9F+XcSrx2w2N97QYCo3eZC+Z90jeBEI/s2feOGwSFry5E+z8WQ3h/byfEINYrR8+XMp6Dk= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPx5MnPAbZWBaZTH296KC5i8OYkcuVnFmXd7j+LaYUDzFTxt5onthKRHp/o9mC4w0ZSVTTYcuaye/XOBDSRIHMU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-27_05:52:49 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-27T06:30:10.153 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa; will attempt to use it 2024-05-27T06:30:10.155 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks 2024-05-27T06:30:10.155 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-27T06:30:10.157 INFO:teuthology.task.internal:Checking packages... 2024-05-27T06:30:10.180 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '38f9d900c83c63799fbdbe61acc9a11b0d3554a6' 2024-05-27T06:30:10.181 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-27T06:30:10.181 INFO:teuthology.packaging:ref: None 2024-05-27T06:30:10.181 INFO:teuthology.packaging:tag: None 2024-05-27T06:30:10.181 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:30:10.181 INFO:teuthology.packaging:sha1: 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:30:10.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:30:10.339 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:30:10.341 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-27T06:30:10.347 INFO:teuthology.task.internal:no buildpackages task found 2024-05-27T06:30:10.347 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-27T06:30:10.353 INFO:teuthology.task.internal:Saving configuration 2024-05-27T06:30:10.371 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-27T06:30:10.377 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-27T06:30:10.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:24:34.988488', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDejHYRQvFi3F6SJu6biH2eOPomuklsVa7iALC1H9V1S'} 2024-05-27T06:30:10.428 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-27 06:24:34.989686', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPfHtN0G9ATfGfonyLyQUp6apfLpNqveIJ91LpXzNASf'} 2024-05-27T06:30:10.429 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-27T06:30:10.435 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-27T06:30:10.435 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-05-27T06:30:10.435 INFO:teuthology.run_tasks:Running task console_log... 2024-05-27T06:30:10.494 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f59a5d1bac0>, signals=[15]) 2024-05-27T06:30:10.494 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-27T06:30:10.501 INFO:teuthology.task.internal:Opening connections... 2024-05-27T06:30:10.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:30:10.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:30:10.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:30:10.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:30:10.660 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-27T06:30:10.666 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-05-27T06:30:10.672 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-05-27T06:30:10.672 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="22.04" 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=jammy 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-27T06:30:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-27T06:30:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-27T06:30:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-27T06:30:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=jammy 2024-05-27T06:30:10.720 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-05-27T06:30:10.740 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-05-27T06:30:10.745 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-05-27T06:30:10.745 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-05-27T06:30:10.794 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-27T06:30:10.794 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Ubuntu" 2024-05-27T06:30:10.794 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="22.04" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_CODENAME=jammy 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:ID=ubuntu 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE=debian 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-27T06:30:10.795 INFO:teuthology.orchestra.run.smithi106.stdout:UBUNTU_CODENAME=jammy 2024-05-27T06:30:10.795 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-05-27T06:30:10.814 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-27T06:30:10.824 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-27T06:30:10.829 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-27T06:30:10.830 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:30:10.832 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-05-27T06:30:10.838 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-27T06:30:10.845 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-27T06:30:10.845 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:30:10.880 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-05-27T06:30:10.888 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-27T06:30:10.943 INFO:teuthology.run_tasks:Running task kernel... 2024-05-27T06:30:10.957 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'} 2024-05-27T06:30:10.957 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'mon.b': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, timeout 300 2024-05-27T06:30:10.957 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:30:10.958 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-27T06:30:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:30:10.962 DEBUG:teuthology.task.kernel:sha1 for mon.a is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:10.963 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:30:10.963 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-05-27T06:30:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:30:10.964 DEBUG:teuthology.task.kernel:sha1 for mon.b is b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:10.964 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "b9459a26b0888785d05c47deb35abc9d0354ab93"... 2024-05-27T06:30:10.964 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-05-27T06:30:11.011 INFO:teuthology.orchestra.run.smithi081.stdout:5.15.0-56-generic 2024-05-27T06:30:11.011 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:11.011 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:30:11.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:11.015 INFO:teuthology.orchestra.run.smithi106.stdout:5.15.0-56-generic 2024-05-27T06:30:11.015 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:11.015 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-27T06:30:11.015 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:30:11.162 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-05-27T06:30:11.162 DEBUG:teuthology.orchestra.run.smithi081:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-05-27T06:30:11.164 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-05-27T06:30:11.164 DEBUG:teuthology.orchestra.run.smithi106:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-05-27T06:30:11.178 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-05-27T06:30:11.186 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: git config --global init.defaultBranch 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: 2024-05-27T06:30:11.323 INFO:teuthology.orchestra.run.smithi081.stderr:hint: git branch -m 2024-05-27T06:30:11.324 INFO:teuthology.orchestra.run.smithi081.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-05-27T06:30:11.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-05-27T06:30:11.340 INFO:teuthology.orchestra.run.smithi106.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-05-27T06:30:11.340 INFO:teuthology.orchestra.run.smithi106.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-05-27T06:30:11.340 INFO:teuthology.orchestra.run.smithi106.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: git config --global init.defaultBranch 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: 2024-05-27T06:30:11.341 INFO:teuthology.orchestra.run.smithi106.stderr:hint: git branch -m 2024-05-27T06:30:11.342 INFO:teuthology.orchestra.run.smithi106.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-05-27T06:30:11.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-05-27T06:30:11.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-05-27T06:30:11.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-05-27T06:30:11.456 DEBUG:teuthology.orchestra.run.smithi081:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-05-27T06:30:11.467 DEBUG:teuthology.orchestra.run.smithi106:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-05-27T06:31:01.894 INFO:teuthology.orchestra.run.smithi081.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-05-27T06:31:01.895 INFO:teuthology.orchestra.run.smithi081.stderr: * [new branch] main -> origin/main 2024-05-27T06:31:01.929 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190312 -> 20190312 2024-05-27T06:31:01.929 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190416 -> 20190416 2024-05-27T06:31:01.929 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190514 -> 20190514 2024-05-27T06:31:01.929 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190618 -> 20190618 2024-05-27T06:31:01.929 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190717 -> 20190717 2024-05-27T06:31:01.930 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190815 -> 20190815 2024-05-27T06:31:01.930 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20190923 -> 20190923 2024-05-27T06:31:01.930 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20191022 -> 20191022 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20191215 -> 20191215 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200122 -> 20200122 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200316 -> 20200316 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200421 -> 20200421 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200519 -> 20200519 2024-05-27T06:31:01.931 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200619 -> 20200619 2024-05-27T06:31:01.932 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200721 -> 20200721 2024-05-27T06:31:01.932 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200817 -> 20200817 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20200918 -> 20200918 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20201022 -> 20201022 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20201118 -> 20201118 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20201218 -> 20201218 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210208 -> 20210208 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210315 -> 20210315 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210511 -> 20210511 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210716 -> 20210716 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210818 -> 20210818 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20210919 -> 20210919 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20211027 -> 20211027 2024-05-27T06:31:01.933 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20211216 -> 20211216 2024-05-27T06:31:01.935 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220209 -> 20220209 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220310 -> 20220310 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220411 -> 20220411 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220509 -> 20220509 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220610 -> 20220610 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220708 -> 20220708 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220815 -> 20220815 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20220913 -> 20220913 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20221012 -> 20221012 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20221109 -> 20221109 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20221214 -> 20221214 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230117 -> 20230117 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230210 -> 20230210 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230310 -> 20230310 2024-05-27T06:31:01.936 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230404 -> 20230404 2024-05-27T06:31:01.937 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230515 -> 20230515 2024-05-27T06:31:01.937 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230625 -> 20230625 2024-05-27T06:31:01.937 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230804 -> 20230804 2024-05-27T06:31:01.937 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20230919 -> 20230919 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20231030 -> 20231030 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20231111 -> 20231111 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20231211 -> 20231211 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20240115 -> 20240115 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20240220 -> 20240220 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20240312 -> 20240312 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20240410 -> 20240410 2024-05-27T06:31:01.938 INFO:teuthology.orchestra.run.smithi081.stderr: * [new tag] 20240513 -> 20240513 2024-05-27T06:31:04.282 INFO:teuthology.orchestra.run.smithi106.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-05-27T06:31:04.282 INFO:teuthology.orchestra.run.smithi106.stderr: * [new branch] main -> origin/main 2024-05-27T06:31:04.311 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190312 -> 20190312 2024-05-27T06:31:04.311 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190416 -> 20190416 2024-05-27T06:31:04.311 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190514 -> 20190514 2024-05-27T06:31:04.311 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190618 -> 20190618 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190717 -> 20190717 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190815 -> 20190815 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20190923 -> 20190923 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20191022 -> 20191022 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20191215 -> 20191215 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200122 -> 20200122 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200316 -> 20200316 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200421 -> 20200421 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200519 -> 20200519 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200619 -> 20200619 2024-05-27T06:31:04.312 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200721 -> 20200721 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200817 -> 20200817 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20200918 -> 20200918 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20201022 -> 20201022 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20201118 -> 20201118 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20201218 -> 20201218 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210208 -> 20210208 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210315 -> 20210315 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210511 -> 20210511 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210716 -> 20210716 2024-05-27T06:31:04.316 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210818 -> 20210818 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20210919 -> 20210919 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20211027 -> 20211027 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20211216 -> 20211216 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220209 -> 20220209 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220310 -> 20220310 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220411 -> 20220411 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220509 -> 20220509 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220610 -> 20220610 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220708 -> 20220708 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220815 -> 20220815 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20220913 -> 20220913 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20221012 -> 20221012 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20221109 -> 20221109 2024-05-27T06:31:04.317 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20221214 -> 20221214 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230117 -> 20230117 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230210 -> 20230210 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230310 -> 20230310 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230404 -> 20230404 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230515 -> 20230515 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230625 -> 20230625 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230804 -> 20230804 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20230919 -> 20230919 2024-05-27T06:31:04.318 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20231030 -> 20231030 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20231111 -> 20231111 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20231211 -> 20231211 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20240115 -> 20240115 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20240220 -> 20240220 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20240312 -> 20240312 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20240410 -> 20240410 2024-05-27T06:31:04.320 INFO:teuthology.orchestra.run.smithi106.stderr: * [new tag] 20240513 -> 20240513 2024-05-27T06:31:06.232 INFO:teuthology.orchestra.run.smithi081.stderr:Updating files: 45% (1550/3421) Updating files: 46% (1574/3421) Updating files: 47% (1608/3421) Updating files: 48% (1643/3421) Updating files: 49% (1677/3421) Updating files: 50% (1711/3421) Updating files: 51% (1745/3421) Updating files: 52% (1779/3421) Updating files: 53% (1814/3421) Updating files: 54% (1848/3421) Updating files: 55% (1882/3421) Updating files: 56% (1916/3421) Updating files: 57% (1950/3421) Updating files: 58% (1985/3421) Updating files: 59% (2019/3421) Updating files: 59% (2043/3421) Updating files: 60% (2053/3421) Updating files: 61% (2087/3421) Updating files: 62% (2122/3421) Updating files: 63% (2156/3421) Updating files: 64% (2190/3421) Updating files: 65% (2224/3421) Updating files: 66% (2258/3421) Updating files: 67% (2293/3421) Updating files: 68% (2327/3421) Updating files: 69% (2361/3421) Updating files: 70% (2395/3421) Updating files: 70% (2402/3421) Updating files: 71% (2429/3421) Updating files: 72% (2464/3421) Updating files: 73% (2498/3421) Updating files: 74% (2532/3421) Updating files: 75% (2566/3421) Updating files: 76% (2600/3421) Updating files: 77% (2635/3421) Updating files: 78% (2669/3421) Updating files: 79% (2703/3421) Updating files: 80% (2737/3421) Updating files: 81% (2772/3421) Updating files: 82% (2806/3421) Updating files: 83% (2840/3421) Updating files: 83% (2846/3421) Updating files: 84% (2874/3421) Updating files: 85% (2908/3421) Updating files: 86% (2943/3421) Updating files: 87% (2977/3421) Updating files: 88% (3011/3421) Updating files: 89% (3045/3421) Updating files: 90% (3079/3421) Updating files: 91% (3114/3421) Updating files: 92% (3148/3421) Updating files: 93% (3182/3421) Updating files: 94% (3216/3421) Updating files: 95% (3250/3421) Updating files: 96% (3285/3421) Updating files: 97% (3319/3421) Updating files: 98% (3353/3421) Updating files: 99% (3387/3421) Updating files: 100% (3421/3421) Updating files: 100% (3421/3421), done. 2024-05-27T06:31:06.237 INFO:teuthology.orchestra.run.smithi081.stdout:HEAD is now at ec8627ef Merge branch 'amd-isp-staging' into 'main' 2024-05-27T06:31:06.259 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:31:06.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:31:06.487 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ 2024-05-27T06:31:06.487 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ --input-file=- 2024-05-27T06:31:06.489 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi081.front.sepia.ceph.com to complete... 2024-05-27T06:31:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27 06:31:07 URL:https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb [44988888/44988888] -> "/tmp/linux-image.deb" [1] 2024-05-27T06:31:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:FINISHED --2024-05-27 06:31:07-- 2024-05-27T06:31:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:Total wall clock time: 1.7s 2024-05-27T06:31:08.180 INFO:teuthology.orchestra.run.smithi081.stderr:Downloaded: 1 files, 43M in 1.6s (27.7 MB/s) 2024-05-27T06:31:08.184 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.a... 2024-05-27T06:31:08.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo dpkg -i /tmp/linux-image.deb 2024-05-27T06:31:08.419 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package linux-image-6.9.0-gb9459a26b088. 2024-05-27T06:31:08.627 INFO:teuthology.orchestra.run.smithi106.stderr:Updating files: 45% (1542/3421) Updating files: 46% (1574/3421) Updating files: 47% (1608/3421) Updating files: 48% (1643/3421) Updating files: 49% (1677/3421) Updating files: 50% (1711/3421) Updating files: 51% (1745/3421) Updating files: 52% (1779/3421) Updating files: 53% (1814/3421) Updating files: 54% (1848/3421) Updating files: 55% (1882/3421) Updating files: 56% (1916/3421) Updating files: 57% (1950/3421) Updating files: 58% (1985/3421) Updating files: 59% (2019/3421) Updating files: 59% (2036/3421) Updating files: 60% (2053/3421) Updating files: 61% (2087/3421) Updating files: 62% (2122/3421) Updating files: 63% (2156/3421) Updating files: 64% (2190/3421) Updating files: 65% (2224/3421) Updating files: 66% (2258/3421) Updating files: 67% (2293/3421) Updating files: 68% (2327/3421) Updating files: 69% (2361/3421) Updating files: 70% (2395/3421) Updating files: 70% (2402/3421) Updating files: 71% (2429/3421) Updating files: 72% (2464/3421) Updating files: 73% (2498/3421) Updating files: 74% (2532/3421) Updating files: 75% (2566/3421) Updating files: 76% (2600/3421) Updating files: 77% (2635/3421) Updating files: 78% (2669/3421) Updating files: 79% (2703/3421) Updating files: 80% (2737/3421) Updating files: 81% (2772/3421) Updating files: 82% (2806/3421) Updating files: 83% (2840/3421) Updating files: 83% (2842/3421) Updating files: 84% (2874/3421) Updating files: 85% (2908/3421) Updating files: 86% (2943/3421) Updating files: 87% (2977/3421) Updating files: 88% (3011/3421) Updating files: 89% (3045/3421) Updating files: 90% (3079/3421) Updating files: 91% (3114/3421) Updating files: 92% (3148/3421) Updating files: 93% (3182/3421) Updating files: 94% (3216/3421) Updating files: 95% (3250/3421) Updating files: 96% (3285/3421) Updating files: 97% (3319/3421) Updating files: 98% (3353/3421) Updating files: 99% (3387/3421) Updating files: 100% (3421/3421) Updating files: 100% (3421/3421), done. 2024-05-27T06:31:08.631 INFO:teuthology.orchestra.run.smithi106.stdout:HEAD is now at ec8627ef Merge branch 'amd-isp-staging' into 'main' 2024-05-27T06:31:08.653 INFO:teuthology.task.kernel:Downloading kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:31:08.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b9459a26b0888785d05c47deb35abc9d0354ab93 2024-05-27T06:31:08.879 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ 2024-05-27T06:31:08.879 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/ --input-file=- 2024-05-27T06:31:08.882 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi106.front.sepia.ceph.com to complete... 2024-05-27T06:31:10.492 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-05-27T06:31:10.496 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-05-27T06:31:10.603 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T06:31:10.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27 06:31:10 URL:https://2.chacra.ceph.com/r/kernel/testing/b9459a26b0888785d05c47deb35abc9d0354ab93/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-gb9459a26b088/linux-image-6.9.0-gb9459a26b088_6.9.0-gb9459a26b088-1_amd64.deb [44988888/44988888] -> "/tmp/linux-image.deb" [1] 2024-05-27T06:31:10.758 INFO:teuthology.orchestra.run.smithi106.stderr:FINISHED --2024-05-27 06:31:10-- 2024-05-27T06:31:10.758 INFO:teuthology.orchestra.run.smithi106.stderr:Total wall clock time: 1.9s 2024-05-27T06:31:10.759 INFO:teuthology.orchestra.run.smithi106.stderr:Downloaded: 1 files, 43M in 1.7s (26.0 MB/s) 2024-05-27T06:31:10.763 INFO:teuthology.task.kernel:Installing kernel b9459a26b0888785d05c47deb35abc9d0354ab93 on mon.b... 2024-05-27T06:31:10.763 DEBUG:teuthology.orchestra.run.smithi106:> sudo dpkg -i /tmp/linux-image.deb 2024-05-27T06:31:10.958 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package linux-image-6.9.0-gb9459a26b088. 2024-05-27T06:31:12.982 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-05-27T06:31:12.987 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-05-27T06:31:13.105 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T06:31:14.453 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T06:31:14.557 INFO:teuthology.orchestra.run.smithi081.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:17.157 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up linux-image-6.9.0-gb9459a26b088 (6.9.0-gb9459a26b088-1) ... 2024-05-27T06:31:17.254 INFO:teuthology.orchestra.run.smithi106.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:36.619 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub' 2024-05-27T06:31:36.642 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T06:31:36.665 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-05-27T06:31:36.976 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:36.995 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:37.290 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T06:31:37.297 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T06:31:37.360 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T06:31:37.366 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T06:31:37.512 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T06:31:37.512 INFO:teuthology.orchestra.run.smithi081.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T06:31:37.512 INFO:teuthology.orchestra.run.smithi081.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T06:31:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-05-27T06:31:37.722 DEBUG:teuthology.orchestra.run.smithi081:> dpkg-deb -c /tmp/linux-image.deb 2024-05-27T06:31:37.729 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./ 2024-05-27T06:31:37.730 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./boot/ 2024-05-27T06:31:37.730 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7004139 2024-05-23 11:07 ./boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:31:37.744 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 172251 2024-05-23 11:07 ./boot/config-6.9.0-gb9459a26b088 2024-05-27T06:31:37.744 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13136896 2024-05-23 11:07 ./boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:37.765 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/ 2024-05-27T06:31:37.765 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postinst.d/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postrm.d/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/preinst.d/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/prerm.d/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ 2024-05-27T06:31:37.766 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 507497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/aesni-intel.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 109897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 155241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 87001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:31:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/ 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/ 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/ 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpuid.ko 2024-05-27T06:31:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/msr.ko 2024-05-27T06:31:37.769 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/ 2024-05-27T06:31:37.769 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 443289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:31:37.769 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1037081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:31:37.769 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 2850153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:31:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ 2024-05-27T06:31:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/af_alg.ko 2024-05-27T06:31:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_hash.ko 2024-05-27T06:31:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_skcipher.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/ 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/ 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:31:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:31:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cryptd.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_simd.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lrw.ko 2024-05-27T06:31:37.783 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 169057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:31:37.784 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xts.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_extlog.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/ 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/einj.ko 2024-05-27T06:31:37.785 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 119017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 99177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 107729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:31:37.786 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:31:37.787 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_legacy.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:31:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_radisys.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rz1000.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:31:37.789 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 102113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ 2024-05-27T06:31:37.790 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/ 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 163833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/ 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/ 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 140833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/ 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1241585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 203281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/ 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 115257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:31:37.791 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 143425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/ 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 126609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 119865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 313209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/ 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 117217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:31:37.839 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/ 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/applicom.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/ 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:31:37.840 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 186913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 179441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/ 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 103385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/nvram.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/ 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:31:37.841 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/ 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_class.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_core.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/ 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-palmas.ko 2024-05-27T06:31:37.842 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-si5351.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-twl6040.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-wm831x.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 89209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 333593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/ 2024-05-27T06:31:37.843 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 751433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ 2024-05-27T06:31:37.844 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 146745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 96137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:31:37.845 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 68689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/ 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-gpio.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max14577.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max77693.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max8997.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-palmas.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-rt8973a.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-sm5502.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/ 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/dmi-sysfs.ko 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/ 2024-05-27T06:31:37.846 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/efi-pstore.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/ 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-adp5520.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-amd8111.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-arizona.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9052.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9055.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-dln2.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 81065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-f7188x.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-generic.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-05-27T06:31:37.847 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-kempld.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-lp3943.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max7300.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max730x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max732x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pca953x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pcf857x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-rdc321x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch311x.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-tps65912.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl4030.ko 2024-05-27T06:31:37.848 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl6040.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-vx855.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm831x.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8350.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8994.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/ 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-a4tech.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-apple.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-belkin.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cherry.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-chicony.ko 2024-05-27T06:31:37.849 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 74385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cp2112.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cypress.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ezkey.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-generic.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:31:37.850 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kensington.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lg-g15.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 121361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:31:37.851 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-magicmouse.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-microsoft.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-monterey.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ntrig.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:31:37.852 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:31:37.853 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 81097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:31:37.854 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 184729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 319897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/ 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 153617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbhid.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbkbd.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbmouse.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 261929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/ 2024-05-27T06:31:37.855 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 75057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 110657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 381625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 77145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 115049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:31:37.856 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 89017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 81465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 68857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:31:37.857 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9052-hwmon.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9055-hwmon.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 113129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc2103.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:31:37.858 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hih6130.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:31:37.859 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 156457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/jc42.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:31:37.860 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 86489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 76481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 131497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:31:37.861 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mc13783-adc.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-05-27T06:31:37.862 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 141345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 72729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:31:37.863 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 101225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:31:37.864 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 71161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:31:37.865 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 96569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 75121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 83417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 116769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 134433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/ 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/ 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:31:37.866 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/ 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:31:37.867 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 76961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:31:37.868 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-05-27T06:31:37.869 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/ 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 329681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1119785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 83897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 408329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:31:37.870 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 120905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 347201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 84937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 590873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 514577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 911649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 375449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 197097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-05-27T06:31:37.871 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 212497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 334353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 120649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 132577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 157929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ 2024-05-27T06:31:37.872 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 156665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/evbug.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ff-memless.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/ 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/ 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-05-27T06:31:37.873 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lkkbd.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8323.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8333.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/newtonkbd.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt1070.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt2160.ko 2024-05-27T06:31:37.874 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/stowaway.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/sunkbd.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/xtkbd.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/matrix-keymap.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/ 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 159993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 114193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:31:37.875 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/gpio_mouse.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 428121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/psmouse.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/ 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 140921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ct82c710.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:31:37.876 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/parkbd.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/pcips2.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ps2mult.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serport.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/ 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-88pm860x.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-adp5520.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 77193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-bd2802.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da903x.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da9052.ko 2024-05-27T06:31:37.877 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-gpio.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3533.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm355x.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3642.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp8788.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-max8997.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-mc13783.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-menf21bmc.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca9532.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca955x.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca963x.ko 2024-05-27T06:31:37.878 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-tca6507.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm831x-status.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm8350.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:31:37.879 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/ 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/ 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 790665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 116817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bufio.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 199017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 134937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-era.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log.ko 2024-05-27T06:31:37.880 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-mirror.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-region-hash.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 139793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-snapshot.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 234441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-zero.ko 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/ 2024-05-27T06:31:37.881 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 264273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 124705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 157673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 444649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/ 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/ 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 214833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 77977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 150753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:31:37.882 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 63865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/ 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm800.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm805.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm80x.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona-i2c.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 148545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/bcm590xx.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/cros_ec_dev.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/dln2.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/janz-cmodio.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/kempld-core.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-core.ko 2024-05-27T06:31:37.883 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lp3943.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/max8907.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-core.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/menf21bmc.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-adc.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-gpio.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/retu-mfd.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 99409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/si476x-core.ko 2024-05-27T06:31:37.884 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps65010.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps6507x.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wm8994.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/ 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:31:37.885 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/ 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/core.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/ 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ds1682.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/dummy-irq.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/ 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/echo.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/ 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:31:37.886 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/ 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 191617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hmc6352.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 92289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-05-27T06:31:37.887 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ics932s401.ko 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/ 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/ 2024-05-27T06:31:37.890 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 154257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 388977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/phantom.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/ 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/st_drv.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:31:37.891 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 234849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 620849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 139745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/ 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 589537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:31:37.892 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/ 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 158641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 109513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/ 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:31:37.893 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/ 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 84321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/ 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/ 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/ 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:31:37.894 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 115873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:31:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/ 2024-05-27T06:31:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/ 2024-05-27T06:31:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 124481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:31:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-05-27T06:31:37.963 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 128393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 110321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/ 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 101849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 88945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/ 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 105625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 228977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1733297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 167177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/ 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 164249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:31:37.964 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 440553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 432145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 257145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 446057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1311193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 206673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:31:37.965 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 269249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 77449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 169041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 76201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/ 2024-05-27T06:31:37.966 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 75937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ec_bhf.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/ 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/ 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 356353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 63177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/ 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 115561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/ 2024-05-27T06:31:37.967 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 366193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 755673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 324513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1196201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 362609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 672313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/ 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 121513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:31:37.968 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 966601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 193465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 126873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 150769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 904929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 344089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-05-27T06:31:37.969 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1413529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 117345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/ 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 131353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/ 2024-05-27T06:31:37.970 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 181201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 171409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 300793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 105681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 885057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/ 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:31:37.971 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 81633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 100681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 264145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 142401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1350201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/ 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:31:37.972 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 80057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/ 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 74281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 79449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 100777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 83433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/ 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 644329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/ 2024-05-27T06:31:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 128489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 207985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 82625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/ 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/ 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 110721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/ 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 98681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 102505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/ 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:31:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/defxx.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 275713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/skfp.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 290441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:31:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/ 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-gpio.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/ 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:31:37.976 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 103369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 96017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:31:37.977 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 121097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/ 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/plip.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:31:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sb1000.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/ 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 80153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/ 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 137313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:31:37.979 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 176481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 384329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 175513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/ 2024-05-27T06:31:37.980 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 335489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 127497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/ 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 168113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 103881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/ 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/ 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pci-stub.ko 2024-05-27T06:31:37.981 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/ 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 179937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_core.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 133745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/ 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/ 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/ 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-05-27T06:31:37.982 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/ 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/ 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/ 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/ 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 87929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:31:37.983 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 164761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ibm_rtl.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/ 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:31:37.984 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/ 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ 2024-05-27T06:31:37.985 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 68825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq2415x_charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 73065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24190_charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24735-charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9030_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9052-battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2760_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2780_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2781_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2782_battery.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/gpio-charger.ko 2024-05-27T06:31:37.986 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/isp1704_charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/lp8727_charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max14577_charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max17042_battery.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8903_charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8925_power.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/pcf50633-charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/sbs-battery.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/test_power.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/tps65090-charger.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_backup.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_power.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm8350_power.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/ 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 73289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:31:37.987 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/ 2024-05-27T06:31:37.988 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/ 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm80x.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm860x.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9052.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9055.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9063.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 82217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:31:37.989 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-lp8788.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t86.ko 2024-05-27T06:31:37.990 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8907.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8925.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8997.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8998.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-palmas.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf50633.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:31:37.991 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rc5t583.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-s35390a.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps6586x.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps65910.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm831x.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm8350.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ 2024-05-27T06:31:37.992 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 164473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/ 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 340689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 120393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/ 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 468049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 380633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/ 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 243649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-05-27T06:31:37.993 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/ 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 135601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/ 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 304969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/ 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1527017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/ 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 297769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/ 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 144377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:31:37.994 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 390657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 88873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 117473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 173009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 267521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/ 2024-05-27T06:31:37.995 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 95321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 194001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fdomain.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/ 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 366457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 251497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/imm.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 327833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 129777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/ 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 397657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:31:37.996 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 63113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/ 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 386801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 178169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/ 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 273953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/ 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 2735913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/ 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 88521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 389089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:31:37.997 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 104761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/ 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 836897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/ 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 158705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 96529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/ 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 101609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/ 2024-05-27T06:31:37.998 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 443713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 152025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ppa.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/ 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 2324713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 103593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/ 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 697281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qlogicfas408.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 322857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 219849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:31:37.999 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 357145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 121177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 93473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 138113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 57393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/ 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 205001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/xen-scsifront.ko 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ 2024-05-27T06:31:38.000 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 174865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/ 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/ 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 920073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/ 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1170873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 149777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/ 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:31:38.001 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/ 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/ 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/ 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:31:38.002 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 99289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/moxa.ko 2024-05-27T06:31:38.003 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/mxser.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 134329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/ 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/ 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_uart.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:31:38.004 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/ 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 120553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/rp2.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/sc16is7xx.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/ 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 693057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/ 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:31:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_dmem_genirq.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_mf624.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_netx.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/ 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/ 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/c67x00.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 189169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-05-27T06:31:38.006 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/ 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 101537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 63017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 107409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ulpi.ko 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/ 2024-05-27T06:31:38.007 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/ 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 575217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/ 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/ 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 72241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_ether.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_serial.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-05-27T06:31:38.008 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 131513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 146601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 101233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 71241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 84929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/ 2024-05-27T06:31:38.009 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-05-27T06:31:38.010 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 204009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/libcomposite.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/ 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/ 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 182537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2272.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2280.ko 2024-05-27T06:31:38.011 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 94833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 80513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 87457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 95433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 195953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/bcma-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/isp116x-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 104433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 102265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811_cs.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ssb-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-05-27T06:31:38.012 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/ 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cytherm.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ehset.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:31:38.013 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/lvstest.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/ 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:31:38.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 84257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbtest.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/ 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 91505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/usbmon.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/ 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 396169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/musb_hdrc.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/ 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-isp1301.ko 2024-05-27T06:31:38.015 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-tahvo.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 58321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/f81232.ko 2024-05-27T06:31:38.016 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 188601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 152297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 142825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 69209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 122321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/metro-usb.ko 2024-05-27T06:31:38.017 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 89961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mxuport.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 197201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:31:38.018 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 151121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usbserial.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/wishbone-serial.ko 2024-05-27T06:31:38.019 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 64041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:31:38.020 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 201145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/ 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 76025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 122233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/ 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/ 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 206897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:31:38.021 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 134241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 93665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/ 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 138641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 71273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 107897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/ 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/ 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/aat2870_bl.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp5520_bl.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8860_bl.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 51737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8870_bl.ko 2024-05-27T06:31:38.022 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/as3711_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/bd6107.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da903x_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da9052_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/gpio_backlight.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/kb3886_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3533_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3639_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lv5207lp.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/max8925_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pandora_bl.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-05-27T06:31:38.023 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/wm831x_bl.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/ 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/matrox_w1.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/w1-gpio.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/ 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:31:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 129409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/acquirewdt.ko 2024-05-27T06:31:38.025 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/advantechwdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/cpu5wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9052_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9055_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9063_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/dw_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/eurotechwdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:31:38.026 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/kempld_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/mena21_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pc87413_wdt.ko 2024-05-27T06:31:38.027 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/retu_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sc1200wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/twl4030_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:31:38.028 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wafer5823wdt.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm831x_wdt.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm8350_wdt.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/ 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/ 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 213865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-scsiback.ko 2024-05-27T06:31:38.029 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 194449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 78049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/adfs.ko 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 197801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/ 2024-05-27T06:31:38.030 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1260937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/kafs.ko 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/ 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 127337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/autofs4.ko 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/ 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 80481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/ 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/bfs.ko 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/ 2024-05-27T06:31:38.031 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 4491713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/ 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 376801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1586617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/ 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 130017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/ 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 139049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/configfs.ko 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/ 2024-05-27T06:31:38.080 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 813137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/efs.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1906713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/freevxfs.ko 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/ 2024-05-27T06:31:38.081 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1486393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 183217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 281993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 248945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/hpfs.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 92985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/ 2024-05-27T06:31:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 532953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 361281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 112569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 591625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 102537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 74337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/ 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 147673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:31:38.083 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1320377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 82089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv2.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 151617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 2518385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/ 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/ 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 2021969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/ 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 733969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/ 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:31:38.084 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ascii.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:31:38.085 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:31:38.086 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 95617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 144201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 167129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 119505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:31:38.087 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:31:38.088 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 634457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ntfs3.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 302033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 548057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 3245449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 22105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:31:38.089 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/ 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/omfs.ko 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/ 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 462817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/ 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/qnx4.ko 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/ 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 56073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/qnx6.ko 2024-05-27T06:31:38.090 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/ 2024-05-27T06:31:38.091 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_tree.ko 2024-05-27T06:31:38.091 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v1.ko 2024-05-27T06:31:38.091 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v2.ko 2024-05-27T06:31:38.094 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/ 2024-05-27T06:31:38.094 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 684713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:31:38.094 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/ 2024-05-27T06:31:38.094 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:31:38.095 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/ 2024-05-27T06:31:38.095 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/ 2024-05-27T06:31:38.095 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 3350841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:31:38.095 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/ 2024-05-27T06:31:38.095 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/ 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 164145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/ 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 115865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:31:38.096 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/ 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 309937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 198185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/ 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 5192193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:31:38.097 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/ 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/ 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 97617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/locktorture.ko 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/ 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/test_udelay.ko 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 72657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:31:38.098 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ 2024-05-27T06:31:38.099 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:31:38.099 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc7.ko 2024-05-27T06:31:38.099 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:31:38.099 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/ 2024-05-27T06:31:38.099 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/interval_tree_test.ko 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/ 2024-05-27T06:31:38.100 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/ 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/memory-notifier-error-inject.ko 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/notifier-error-inject.ko 2024-05-27T06:31:38.101 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/percpu_test.ko 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/pm-notifier-error-inject.ko 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/ 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 151945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/rbtree_test.ko 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/ 2024-05-27T06:31:38.102 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test-kstrtox.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 4912433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_bpf.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 61161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_firmware.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_module.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_user_copy.ko 2024-05-27T06:31:38.103 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/ 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/xz_dec_test.ko 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/ 2024-05-27T06:31:38.104 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/ 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:31:38.105 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:31:38.106 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:31:38.106 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:31:38.106 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:31:38.106 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:31:38.106 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/ 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/ 2024-05-27T06:31:38.107 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 106697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:31:38.108 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/ 2024-05-27T06:31:38.108 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 236769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:31:38.108 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:31:38.108 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:31:38.108 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/ 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 879593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/ 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1031177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:31:38.109 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:31:38.110 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:31:38.111 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:31:38.112 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:31:38.113 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/ 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1243369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/ 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 153793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/ 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 244825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:31:38.114 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 71025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 74985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/ 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 507649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:31:38.115 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:31:38.116 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:31:38.116 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:31:38.116 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/ 2024-05-27T06:31:38.116 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 128577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/hsr.ko 2024-05-27T06:31:38.116 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 357369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 59385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ 2024-05-27T06:31:38.117 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 75481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:31:38.118 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 77713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:31:38.119 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ip_tables.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:31:38.120 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:31:38.121 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:31:38.122 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:31:38.123 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:31:38.124 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:31:38.125 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 54425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ 2024-05-27T06:31:38.126 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 47425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 103169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_gre.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 118873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 71081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:31:38.127 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/mip6.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:31:38.128 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:31:38.129 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:31:38.130 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 29809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:31:38.131 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 94081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 30289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/ 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 106361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/ 2024-05-27T06:31:38.132 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 153209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 66233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/ 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 63321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/lapb.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/ 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 332817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc2.ko 2024-05-27T06:31:38.133 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/ 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 388817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/ 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 147729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 124033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 124337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 126673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:31:38.134 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 126465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 137825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 72345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 133673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 139065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 138257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 136169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 139889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 534033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:31:38.135 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 55209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 496713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:31:38.136 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 134361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 140793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 79865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 117385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 46097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:31:38.137 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 118337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 811705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:31:38.138 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 65137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 50649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_osf.ko 2024-05-27T06:31:38.139 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 24985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_synproxy.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tproxy.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 108385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/x_tables.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 21857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:31:38.140 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:31:38.141 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:31:38.142 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 52809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:31:38.143 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 16153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:31:38.144 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 11873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpudp.ko 2024-05-27T06:31:38.145 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/ 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 17225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/ 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/ 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 488457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/ 2024-05-27T06:31:38.146 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/ 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 141993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/phonet.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 68801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/pn_pep.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/ 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 318537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 202249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 85073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/ 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1028129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/rxrpc.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/ 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 28713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:31:38.147 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 53457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 45665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 41449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_cgroup.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 37297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 49313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 79137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:31:38.148 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 48777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 25089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 43145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 39169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq_codel.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 62809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:31:38.149 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 81825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 36617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 44785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 32825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 13497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 60241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 33425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:31:38.150 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 35769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 34153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/ 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1187153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/ 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/ 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 432225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1940753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/ 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 889729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/ 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 1122273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:31:38.151 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/ 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 18793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/ 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 105601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 90249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 123745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/ 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 165777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/x25.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/ 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_algo.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:31:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 131857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_user.ko 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 646387 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 630543 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 7706 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 23609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 9941 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 70990 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 146985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 232892 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 210 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 72627 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 879 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 258450 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 313019 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:31:38.153 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/ 2024-05-27T06:31:38.154 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/ 2024-05-27T06:31:38.154 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/ 2024-05-27T06:31:38.154 INFO:teuthology.orchestra.run.smithi081.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/ 2024-05-27T06:31:38.154 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 171 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/changelog.Debian.gz 2024-05-27T06:31:38.154 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r-- root/root 691 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/copyright 2024-05-27T06:31:38.159 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:31:38.159 INFO:teuthology.task.kernel:searching for kernel 6.9.0-gb9459a26b088 2024-05-27T06:31:38.159 DEBUG:teuthology.orchestra.run.smithi081:> egrep '(submenu|menuentry.*6.9.0-gb9459a26b088).*{' /boot/grub/grub.cfg 2024-05-27T06:31:38.165 INFO:teuthology.orchestra.run.smithi081.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:38.166 INFO:teuthology.orchestra.run.smithi081.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:38.166 INFO:teuthology.orchestra.run.smithi081.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:38.166 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-05-27T06:31:38.166 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-gb9459a26b088 2024-05-27T06:31:38.167 DEBUG:teuthology.orchestra.run.smithi081:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-05-27T06:31:38.211 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi081.front.sepia.ceph.com to complete... 2024-05-27T06:31:38.638 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub' 2024-05-27T06:31:38.639 INFO:teuthology.orchestra.run.smithi081.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T06:31:38.667 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-05-27T06:31:38.842 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:38.848 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:38.879 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub' 2024-05-27T06:31:38.892 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T06:31:38.927 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-05-27T06:31:39.059 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T06:31:39.065 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T06:31:39.104 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T06:31:39.109 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T06:31:39.220 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:39.235 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:39.250 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T06:31:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T06:31:39.251 INFO:teuthology.orchestra.run.smithi081.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T06:31:39.268 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-05-27T06:31:39.526 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T06:31:39.536 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T06:31:39.603 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T06:31:39.608 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T06:31:39.760 INFO:teuthology.orchestra.run.smithi106.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T06:31:39.760 INFO:teuthology.orchestra.run.smithi106.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T06:31:39.760 INFO:teuthology.orchestra.run.smithi106.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T06:31:39.803 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-05-27T06:31:39.962 DEBUG:teuthology.orchestra.run.smithi106:> dpkg-deb -c /tmp/linux-image.deb 2024-05-27T06:31:39.970 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./ 2024-05-27T06:31:39.970 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./boot/ 2024-05-27T06:31:39.970 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7004139 2024-05-23 11:07 ./boot/System.map-6.9.0-gb9459a26b088 2024-05-27T06:31:39.984 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 172251 2024-05-23 11:07 ./boot/config-6.9.0-gb9459a26b088 2024-05-27T06:31:39.985 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13136896 2024-05-23 11:07 ./boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:40.007 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/ 2024-05-27T06:31:40.007 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/ 2024-05-27T06:31:40.007 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postinst.d/ 2024-05-27T06:31:40.007 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/postrm.d/ 2024-05-27T06:31:40.007 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/preinst.d/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./etc/kernel/prerm.d/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 507497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/aesni-intel.ko 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-27T06:31:40.008 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 109897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 155241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 87001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-27T06:31:40.009 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/ 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/ 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/ 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/cpuid.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kernel/msr.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/ 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 443289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-27T06:31:40.010 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1037081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-27T06:31:40.011 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 2850153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/arch/x86/kvm/kvm.ko 2024-05-27T06:31:40.020 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/af_alg.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_hash.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/algif_skcipher.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ansi_cprng.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/anubis.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/arc4.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/ 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/ 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_memcpy.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_pq.ko 2024-05-27T06:31:40.021 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_tx.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/async_xor.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/async_tx/raid6test.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authenc.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/authencesn.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blake2b_generic.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_common.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/blowfish_generic.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/camellia_generic.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast5_generic.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast6_generic.ko 2024-05-27T06:31:40.022 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cast_common.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/ccm.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cmac.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crc32_generic.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cryptd.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_engine.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_simd.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/crypto_user.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/cts.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/des_generic.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/dh_generic.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/echainiv.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/essiv.ko 2024-05-27T06:31:40.023 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/fcrypt.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/khazad.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lrw.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/lz4hc.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/md4.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/michael_mic.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcbc.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/pcrypt.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/rmd160.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/seed.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/serpent_generic.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 169057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tcrypt.ko 2024-05-27T06:31:40.024 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/tea.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_common.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/twofish_generic.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/vmac.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/wp512.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xcbc.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xor.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xts.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/crypto/xxhash_generic.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_extlog.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-27T06:31:40.025 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/acpi_pad.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/ 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/apei/einj.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/ec_sys.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/platform_profile.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbs.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/sbshc.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 119017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/acpi/video.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/acard-ahci.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 99177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/ahci_platform.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 107729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci.ko 2024-05-27T06:31:40.026 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/libahci_platform.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_acpi.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ali.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_amd.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_artop.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atiixp.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_atp867x.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd640.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cmd64x.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_cypress.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_efar.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt366.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt37x.ko 2024-05-27T06:31:40.027 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it8213.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_it821x.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_jmicron.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_legacy.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_marvell.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_mpiix.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_netcell.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ninja32.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87410.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_ns87415.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_oldpiix.ko 2024-05-27T06:31:40.028 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_opti.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_optidma.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pcmcia.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_piccolo.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_radisys.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rdc.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_rz1000.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sch.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_serverworks.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sil680.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_sl82c105.ko 2024-05-27T06:31:40.029 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_triflex.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pata_via.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/pdc_adma.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_inic162x.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 102113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_mv.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_nv.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_promise.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_qstor.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sil24.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sis.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_svw.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_sx4.ko 2024-05-27T06:31:40.030 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_uli.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_via.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ata/sata_vsc.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/auxdisplay/ks0108.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/ 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 163833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/bcma/bcma.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/ 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/ 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 140833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/aoe/aoe.ko 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/ 2024-05-27T06:31:40.031 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1241585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/drbd/drbd.ko 2024-05-27T06:31:40.084 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 203281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/floppy.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/ 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 115257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 143425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/nbd.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/ 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 126609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/null_blk/null_blk.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 119865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/pktcdvd.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 313209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/rbd.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/ 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 117217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/ 2024-05-27T06:31:40.085 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/block/zram/zram.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/applicom.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hangcheck-timer.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/ 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/via-rng.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 186913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-27T06:31:40.086 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 179441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/lp.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/ 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 103385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/mwave/mwave.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/nvram.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/ppdev.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tlclk.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/ 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-05-27T06:31:40.087 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/ 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_class.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_core.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/ 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-palmas.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-si5351.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-twl6040.ko 2024-05-27T06:31:40.088 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/clk/clk-wm831x.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 89209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 333593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/ccp/ccp.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/ 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 751433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-27T06:31:40.089 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-aes.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/padlock-sha.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dca/dca.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 146745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-27T06:31:40.090 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 96137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/amd64_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/e752x_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/edac_mce_amd.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3000_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i3200_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5100_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i5400_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7300_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i7core_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/i82975x_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/ie31200_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 68689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/sb_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/edac/x38_edac.ko 2024-05-27T06:31:40.091 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/ 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-gpio.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max14577.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max77693.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-max8997.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-palmas.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-rt8973a.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/extcon/extcon-sm5502.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/ 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/dmi-sysfs.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/ 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/efi/efi-pstore.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-27T06:31:40.092 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/ 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-adp5520.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-amd8111.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-arizona.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9052.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-da9055.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-dln2.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 81065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-f7188x.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-generic.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ich.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-kempld.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-lp3943.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max7300.ko 2024-05-27T06:31:40.093 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max730x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-max732x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pca953x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-pcf857x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-rdc321x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-sch311x.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-tps65912.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl4030.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-twl6040.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-27T06:31:40.094 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-vx855.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm831x.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8350.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/gpio/gpio-wm8994.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/ 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-a4tech.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-apple.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-appleir.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-aureal.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-axff.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-belkin.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cherry.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-chicony.ko 2024-05-27T06:31:40.095 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 74385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cp2112.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-cypress.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-dr.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elecom.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-elo.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-emsff.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ezkey.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gaff.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-generic.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gt683r.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-gyration.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-27T06:31:40.096 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-holtekff.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-hyperv.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-icade.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kensington.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-keytouch.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-kye.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lcpower.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-led.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lenovo.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-lg-g15.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 121361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-logitech.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-magicmouse.ko 2024-05-27T06:31:40.097 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-microsoft.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-monterey.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-multitouch.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ntrig.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-ortek.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-penmount.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-petalynx.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-picolcd.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-pl.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-plantronics.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-primax.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-rmi.ko 2024-05-27T06:31:40.098 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-common.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-roccat.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-saitek.ko 2024-05-27T06:31:40.099 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-samsung.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sjoy.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sony.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-speedlink.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-steelseries.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-sunplus.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tivo.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-tmff.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-topseed.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-twinhan.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 81097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-uclogic.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-waltop.ko 2024-05-27T06:31:40.100 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 184729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-wiimote.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-xinmo.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zpff.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid-zydacron.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 319897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/hid.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/uhid.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/ 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 153617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbhid.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbkbd.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/usbhid/usbmouse.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 261929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hid/wacom.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/ 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 75057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_balloon.ko 2024-05-27T06:31:40.101 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 110657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_utils.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 381625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hv/hv_vmbus.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 77145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/abituguru3.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7414.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ad7418.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adc128d818.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1025.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 115049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1026.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1029.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm1031.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adm9240.ko 2024-05-27T06:31:40.102 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ads7828.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7410.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7411.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 89017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7462.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7470.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 81465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7475.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/adt7x10.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/amc6821.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 68857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/applesmc.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asb100.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asc7621.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/atxp1.ko 2024-05-27T06:31:40.103 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/coretemp.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9052-hwmon.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/da9055-hwmon.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 113129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/dme1737.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds1621.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ds620.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc1403.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc2103.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/emc6w201.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71805f.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f71882fg.ko 2024-05-27T06:31:40.104 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/f75375s.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fam15h_power.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/fschmd.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g760a.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/g762.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl518sm.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/gl520sm.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hih6130.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5500_temp.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/i5k_amb.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmaem.ko 2024-05-27T06:31:40.105 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ibmpex.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina209.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ina2xx.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 156457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/it87.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/jc42.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k10temp.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/k8temp.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lineage-pem.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm63.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm73.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm75.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm77.ko 2024-05-27T06:31:40.106 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm78.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm80.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm83.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 86489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm85.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm87.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 76481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm90.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm92.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 131497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm93.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95234.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95241.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/lm95245.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc2945.ko 2024-05-27T06:31:40.107 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4151.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4215.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4222.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4245.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4260.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/ltc4261.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max16065.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1619.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max1668.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max197.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6639.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6650.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/max6697.ko 2024-05-27T06:31:40.108 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mc13783-adc.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/mcp3021.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6683.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 141345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775-core.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 72729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct6775.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/nct7802.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87360.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pc87427.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pcf8591.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-27T06:31:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 101225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/powr1220.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5627.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch5636.ko 2024-05-27T06:31:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht15.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sht21.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/shtc1.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/sis5595.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47b397.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m1.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/smsc47m192.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/thmc50.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp102.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp103.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp401.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/tmp421.ko 2024-05-27T06:31:40.111 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via-cputemp.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/via686a.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt1211.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/vt8231.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 71161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627ehf.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 96569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83627hf.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83781d.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 75121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83791d.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 83417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83792d.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 116769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83793.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 134433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83795.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l785ts.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/w83l786ng.ko 2024-05-27T06:31:40.112 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/ 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/ 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/ 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-27T06:31:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 76961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-05-27T06:31:40.114 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-05-27T06:31:40.115 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-mux.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-smbus.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/i2c-stub.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/ 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ 2024-05-27T06:31:40.116 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 329681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1119785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_core.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 83897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 408329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 120905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 347201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 84937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/ 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 590873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/ 2024-05-27T06:31:40.117 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 514577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 911649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 375449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 197097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 212497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 334353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ 2024-05-27T06:31:40.118 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 120649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 132577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 157929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 156665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/evbug.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/ff-memless.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/ 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/gameport/gameport.ko 2024-05-27T06:31:40.119 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/joydev.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/ 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lkkbd.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8323.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/lm8333.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-05-27T06:31:40.120 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/newtonkbd.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt1070.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/qt2160.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/stowaway.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/sunkbd.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/keyboard/xtkbd.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/matrix-keymap.ko 2024-05-27T06:31:40.121 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/ 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/appletouch.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/bcm5974.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 159993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/cyapatp.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 114193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/gpio_mouse.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 428121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/psmouse.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/sermouse.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/ 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 140921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-27T06:31:40.122 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/altera_ps2.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/arc_ps2.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ct82c710.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/parkbd.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/pcips2.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/ps2mult.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serio_raw.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/serio/serport.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/input/sparse-keymap.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/ 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-88pm860x.ko 2024-05-27T06:31:40.123 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-adp5520.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 77193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-bd2802.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-blinkm.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da903x.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-da9052.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-gpio.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3530.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3533.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm355x.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lm3642.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp3944.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-lp8788.ko 2024-05-27T06:31:40.124 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-max8997.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-mc13783.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-menf21bmc.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca9532.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca955x.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-pca963x.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-ss4200.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-tca6507.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm831x-status.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/leds-wm8350.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-27T06:31:40.125 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/ 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/ 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 790665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/bcache/bcache.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bio-prison.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 116817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-bufio.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache-smq.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 199017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-cache.ko 2024-05-27T06:31:40.126 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 134937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-crypt.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-delay.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-era.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-flakey.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log-userspace.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-log.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-mirror.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-multipath.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-queue-length.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-raid.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-region-hash.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-round-robin.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-service-time.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 139793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-snapshot.ko 2024-05-27T06:31:40.127 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-switch.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 234441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-thin-pool.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-verity.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/dm-zero.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/ 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 264273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid0.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 124705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid1.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 157673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid10.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 444649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/md/raid456.ko 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/ 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/ 2024-05-27T06:31:40.128 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 214833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptbase.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 77977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptctl.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptfc.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptlan.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 150753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptsas.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptscsih.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 63865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/message/fusion/mptspi.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/ 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm800.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm805.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/88pm80x.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona-i2c.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 148545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/arizona.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/bcm590xx.ko 2024-05-27T06:31:40.129 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/cros_ec_dev.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/dln2.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/janz-cmodio.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/kempld-core.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-core.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lp3943.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_ich.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/lpc_sch.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/max8907.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-core.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-05-27T06:31:40.130 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/menf21bmc.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-adc.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633-gpio.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/pcf50633.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/retu-mfd.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 99409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/si476x-core.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/sm501.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps65010.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/tps6507x.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/viperboard.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/vx855.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wl1273-core.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/mfd/wm8994.ko 2024-05-27T06:31:40.131 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ad525x_dpot.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/ 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds9802als.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/apds990x.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/bh1770glc.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/ 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/c2port/core.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/ 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/cb710/cb710.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ds1682.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/dummy-irq.ko 2024-05-27T06:31:40.132 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/ 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/echo/echo.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/ 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/at24.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/eeprom/max6875.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/enclosure.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/ 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 191617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hmc6352.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/hpilo.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 92289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-05-27T06:31:40.133 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ics932s401.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29003.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/isl29020.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/ 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/ 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 154257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-me.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei-txe.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 388977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/mei/mei.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/phantom.ko 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/ 2024-05-27T06:31:40.134 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/ti-st/st_drv.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_7xx1.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tifm_core.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/tsl2550.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_balloon.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/ 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 234849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/ 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 620849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/bonding/bonding.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/ 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/ 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 139745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-27T06:31:40.135 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/ 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 589537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/dummy.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/eql.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/ 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 158641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-27T06:31:40.136 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 109513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/ 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/ 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 84321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/ 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/ 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/ 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-27T06:31:40.137 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 115873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/ 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/ 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 124481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 128393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 110321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/ 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 101849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 88945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-27T06:31:40.138 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/ 2024-05-27T06:31:40.139 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 105625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-27T06:31:40.139 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 228977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-27T06:31:40.139 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-05-27T06:31:40.139 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1733297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 167177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/ 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 164249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 440553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/ 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/ 2024-05-27T06:31:40.211 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 432145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 257145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 446057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1311193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 206673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/ 2024-05-27T06:31:40.212 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 269249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/ 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/ 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 77449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 169041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 76201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/ 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 75937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-27T06:31:40.213 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/dnet.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ec_bhf.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/ 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/ 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 356353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ethoc.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 63177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fealnx.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/ 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 115561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/ 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 366193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-27T06:31:40.214 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 755673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 324513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1196201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 362609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 672313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 121513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-05-27T06:31:40.215 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 966601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 193465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/jme.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/ 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 126873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 150769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/ 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 904929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 344089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-27T06:31:40.216 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1413529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 117345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 131353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-27T06:31:40.217 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 181201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 171409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 300793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 105681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-05-27T06:31:40.218 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 885057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/ 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/ 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 81633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 100681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 264145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/ 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 142401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/ 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1350201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-27T06:31:40.219 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/ 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/ 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 80057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/ 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 74281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 79449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 100777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 83433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/ 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-27T06:31:40.220 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 644329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/ 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 128489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 207985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 82625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/ 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/ 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 110721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/ 2024-05-27T06:31:40.221 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 98681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 102505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/ 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/ 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/ 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/defxx.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/ 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 275713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/fddi/skfp/skfp.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/geneve.ko 2024-05-27T06:31:40.222 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/ 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 290441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/ 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ifb.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvlan.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/macvtap.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/ 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-gpio.ko 2024-05-27T06:31:40.223 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mdio.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/mii.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/netconsole.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/nlmon.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/ 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/amd.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/bcm87xx.ko 2024-05-27T06:31:40.224 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/broadcom.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/cicada.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/davicom.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/et1011c.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/icplus.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/lxt.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 103369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/marvell.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 96017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/micrel.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/national.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 121097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/phylink.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/qsemi.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/realtek.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/smsc.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/ste10Xp.ko 2024-05-27T06:31:40.225 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/phy/vitesse.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/ 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/plip/plip.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_async.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppoe.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pppox.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/ppp/pptp.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sb1000.ko 2024-05-27T06:31:40.226 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/ 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/slip/slip.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/sungem_phy.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 80153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/tap.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/ 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 137313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_random.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/veth.ko 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/ 2024-05-27T06:31:40.227 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 176481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 384329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/vxlan/vxlan.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 175513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ntb/ntb.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 335489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme-core.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 127497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/nvme/host/nvme.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/ 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 168113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_cs.ko 2024-05-27T06:31:40.228 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 103881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_pc.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/parport/parport_serial.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/ 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/ 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/pci-stub.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pci/xen-pcifront.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/ 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/i82092.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 179937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_core.ko 2024-05-27T06:31:40.229 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/pd6729.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 133745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/ 2024-05-27T06:31:40.230 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 87929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/acerhdf.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 73857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-27T06:31:40.231 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 164761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/hdaps.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/ibm_rtl.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/ 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/intel_ips.ko 2024-05-27T06:31:40.232 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/platform/x86/wmi.ko 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/ 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ 2024-05-27T06:31:40.233 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/88pm860x_charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 68825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq2415x_charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 73065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24190_charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/bq24735-charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9030_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/da9052-battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2760_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2780_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2781_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/ds2782_battery.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/gpio-charger.ko 2024-05-27T06:31:40.234 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/isp1704_charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/lp8727_charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max14577_charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max17042_battery.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8903_charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/max8925_power.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/pcf50633-charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/sbs-battery.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/test_power.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/tps65090-charger.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_backup.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm831x_power.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/power/supply/wm8350_power.ko 2024-05-27T06:31:40.235 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/ 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 73289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/ 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/ 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/clients/pps_parport.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/pps/pps_core.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ptp/ptp_kvm.ko 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/ 2024-05-27T06:31:40.236 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/ 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/ 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ras/amd/fmpm.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/ 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm80x.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-88pm860x.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9052.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9055.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-da9063.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 82217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-27T06:31:40.237 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-em3027.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-lp8788.ko 2024-05-27T06:31:40.238 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-m48t86.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max6900.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8907.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8925.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8997.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-max8998.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-palmas.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf50633.ko 2024-05-27T06:31:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rc5t583.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-s35390a.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps6586x.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-tps65910.ko 2024-05-27T06:31:40.240 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm831x.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-wm8350.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/rtc/rtc-x1205.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-9xxx.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-sas.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/3w-xxxx.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 164473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/BusLogic.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/a100u2w.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/ 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 340689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 120393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/advansys.ko 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/ 2024-05-27T06:31:40.241 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 468049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 380633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/ 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 243649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/am53c974.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/ 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 135601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/atp870u.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/ 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 304969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/ 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1527017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bfa/bfa.ko 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/ 2024-05-27T06:31:40.242 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 297769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/ 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 144377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ch.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/ 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 390657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/ 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 88873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 117473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 173009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dc395x.ko 2024-05-27T06:31:40.243 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/dmx3191d.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/ 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 267521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/esp_scsi.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/ 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 95321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 194001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fdomain.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/ 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 366457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/fnic/fnic.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 251497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hpsa.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hptiop.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/hv_storvsc.ko 2024-05-27T06:31:40.244 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/imm.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/initio.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 327833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ipr.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 129777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ips.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/ 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 397657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/isci/isci.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 63113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/ 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 386801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libfc/libfc.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 178169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/ 2024-05-27T06:31:40.245 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 273953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/libsas/libsas.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/ 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 2735913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/ 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 88521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 389089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 104761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/megaraid.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/ 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 836897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/ 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 158705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 96529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/mvumi.ko 2024-05-27T06:31:40.246 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/ 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 101609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/ 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 443713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 152025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/pmcraid.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ppa.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla1280.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/ 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 2324713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 103593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-27T06:31:40.247 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/ 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 697281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/qlogicfas408.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/raid_class.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 322857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_debug.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 219849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 357145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 121177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 93473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/ses.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 138113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/st.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 57393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/stex.ko 2024-05-27T06:31:40.248 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/ 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 205001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/virtio_scsi.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/wd719x.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/scsi/xen-scsifront.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 174865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ssb/ssb.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/ 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/ 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 920073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/ 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-27T06:31:40.249 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_file.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_iblock.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1170873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_mod.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_pscsi.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 149777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/target_core_user.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/ 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/ 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/ 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/ 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-27T06:31:40.250 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-05-27T06:31:40.251 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 99289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/moxa.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/mxser.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 134329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_gsm.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/n_hdlc.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/nozomi.ko 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/ 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/ 2024-05-27T06:31:40.252 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/altera_uart.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/arc_uart.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/ 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 120553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/rp2.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/tty/serial/sc16is7xx.ko 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/ 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ 2024-05-27T06:31:40.253 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 693057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/ 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_aec.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_cif.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_dmem_genirq.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_mf624.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_netx.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pci_generic.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-05-27T06:31:40.254 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/uio/uio_sercos3.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/ 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/ 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/c67x00/c67x00.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 189169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/ 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 101537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-acm.ko 2024-05-27T06:31:40.255 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 63017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usblp.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 107409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/class/usbtmc.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/common/ulpi.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/ 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/ 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 575217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/dwc3/dwc3.ko 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/ 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/ 2024-05-27T06:31:40.256 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 72241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_ether.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/u_serial.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 131513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 146601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-05-27T06:31:40.257 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 101233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 71241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 84929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/ 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-05-27T06:31:40.258 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 204009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/libcomposite.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/ 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/ 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 182537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-05-27T06:31:40.259 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2272.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/net2280.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 94833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 80513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 87457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 95433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 195953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/bcma-hcd.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/isp116x-hcd.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 104433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-05-27T06:31:40.260 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 102265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/sl811_cs.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/ssb-hcd.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/ 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/mdc800.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/image/microtek.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/adutux.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/appledisplay.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/cytherm.ko 2024-05-27T06:31:40.261 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ehset.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi26.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/emi62.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ezusb.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/idmouse.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/iowarrior.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/ldusb.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/legousbtower.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/lvstest.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/ 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-27T06:31:40.262 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usb3503.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usblcd.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 84257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/usbtest.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/uss720.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/misc/yurex.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/ 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 91505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/mon/usbmon.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/ 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 396169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/musb/musb_hdrc.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/ 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-generic.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-isp1301.ko 2024-05-27T06:31:40.263 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/phy/phy-tahvo.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/aircable.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ark3116.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ch341.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cp210x.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cyberjack.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 58321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/empeg.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/f81232.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 188601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-27T06:31:40.264 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 152297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 142825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/io_ti.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 69209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipaq.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ipw.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ir-usb.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 122321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mct_u232.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/metro-usb.ko 2024-05-27T06:31:40.265 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 79673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7720.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 89961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mos7840.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/mxuport.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/navman.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/omninet.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/opticon.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 197201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/option.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/oti6858.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/pl2303.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcaux.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/qcserial.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/quatech2.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/safe_serial.ko 2024-05-27T06:31:40.266 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/sierra.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ssu100.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/symbolserial.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_debug.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 151121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/usbserial.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/visor.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/whiteheat.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/wishbone-serial.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-27T06:31:40.267 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 64041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/uas.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-karma.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-27T06:31:40.268 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 201145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/storage/usb-storage.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/ 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 76025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 122233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/ 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/ 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 206897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 134241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 93665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-27T06:31:40.269 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/ 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 138641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 71273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_net.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 107897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/vhost/vhost_scsi.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/ 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/ 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/aat2870_bl.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp5520_bl.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8860_bl.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 51737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/adp8870_bl.ko 2024-05-27T06:31:40.270 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/apple_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/as3711_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/bd6107.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da903x_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/da9052_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/gpio_backlight.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/kb3886_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lcd.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3533_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lm3639_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/lv5207lp.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/max8925_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pandora_bl.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-05-27T06:31:40.271 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/video/backlight/wm831x_bl.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/ 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2482.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/ds2490.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/matrox_w1.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/masters/w1-gpio.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/ 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-27T06:31:40.272 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 129409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/w1/wire.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/acquirewdt.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/advantechwdt.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/cpu5wdt.ko 2024-05-27T06:31:40.273 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9052_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9055_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/da9063_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/dw_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/eurotechwdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/hpwdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/i6300esb.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-27T06:31:40.274 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ib700wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ibmasr.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/it87_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/kempld_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/machzwd.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/mena21_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/nv_tco.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pc87413_wdt.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-27T06:31:40.275 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/retu_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sc1200wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/softdog.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/twl4030_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/via_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-27T06:31:40.276 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wafer5823wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wdt_pci.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm831x_wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/wm8350_wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/watchdog/xen_wdt.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/ 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-evtchn.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntalloc.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-gntdev.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/ 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 213865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-27T06:31:40.277 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-privcmd.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xen-scsiback.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23417 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 194449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/9p/9p.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 78049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/adfs/adfs.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 197801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/affs/affs.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/ 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1260937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/afs/kafs.ko 2024-05-27T06:31:40.278 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 127337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/autofs/autofs4.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 80481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/befs/befs.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/bfs/bfs.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/binfmt_misc.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 4491713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/btrfs/btrfs.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 376801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cachefiles/cachefiles.ko 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ 2024-05-27T06:31:40.279 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1586617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ceph/ceph.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 130017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/coda/coda.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 139049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/configfs/configfs.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/cramfs/cramfs.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 813137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/dlm/dlm.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/efs/efs.ko 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/ 2024-05-27T06:31:40.333 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1906713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/f2fs/f2fs.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fat/msdos.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/freevxfs/freevxfs.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/fuse/cuse.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1486393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/gfs2/gfs2.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 183217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfs/hfs.ko 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/ 2024-05-27T06:31:40.334 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 281993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hfsplus/hfsplus.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 248945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/hpfs/hpfs.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 92985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/isofs/isofs.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 532953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/jfs/jfs.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 361281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/lockd/lockd.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 112569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/minix/minix.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/ 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 591625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/netfs/netfs.ko 2024-05-27T06:31:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/ 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/ 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 102537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/ 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 74337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/ 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 147673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1320377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfs.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 82089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv2.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 151617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv3.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 2518385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs/nfsv4.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/ 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/grace.ko 2024-05-27T06:31:40.336 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfs_common/nfs_acl.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/ 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 2021969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nfsd/nfsd.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/ 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 733969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nilfs2/nilfs2.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/ 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-celtic.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-centeuro.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-croatian.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-cyrillic.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-gaelic.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-greek.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-iceland.ko 2024-05-27T06:31:40.337 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-inuit.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-roman.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-romanian.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/mac-turkish.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ascii.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1250.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1251.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp1255.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp737.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp775.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp850.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp852.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp855.ko 2024-05-27T06:31:40.338 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp857.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp860.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp861.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp862.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp863.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp864.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp865.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp866.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp869.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp874.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 95617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp932.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 144201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp936.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 167129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp949.ko 2024-05-27T06:31:40.339 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 119505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_cp950.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_euc-jp.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-1.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-13.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-14.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-15.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-2.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-3.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-4.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-5.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-6.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-7.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_iso8859-9.ko 2024-05-27T06:31:40.340 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-r.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-ru.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_koi8-u.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/nls/nls_utf8.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 634457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ntfs3/ntfs3.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 302033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 548057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ 2024-05-27T06:31:40.341 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 3245449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 22105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/ 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/omfs/omfs.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/ 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 462817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/overlayfs/overlay.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 67961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/pstore/ramoops.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/ 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx4/qnx4.ko 2024-05-27T06:31:40.342 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 56073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/qnx6/qnx6.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_tree.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v1.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/quota/quota_v2.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 684713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/reiserfs/reiserfs.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/romfs/romfs.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/ 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 3350841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/client/cifs.ko 2024-05-27T06:31:40.343 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_arc4.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/smb/common/cifs_md4.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 164145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/squashfs/squashfs.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 115865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/sysv/sysv.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 309937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/udf/udf.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 198185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/ufs/ufs.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 5192193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/fs/xfs/xfs.ko 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/ 2024-05-27T06:31:40.344 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/ 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 97617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/locking/locktorture.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/ 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/time/test_udelay.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 72657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/kernel/torture.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc-itu-t.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc7.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crc8.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/ 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libarc4.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/crypto/libdes.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/interval_tree_test.ko 2024-05-27T06:31:40.345 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/libcrc32c.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lru_cache.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/ 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4_compress.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/lz4/lz4hc_compress.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/ 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/math/cordic.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/memory-notifier-error-inject.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/notifier-error-inject.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/percpu_test.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/pm-notifier-error-inject.ko 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/ 2024-05-27T06:31:40.346 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 151945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/raid6/raid6_pq.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/rbtree_test.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/ 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test-kstrtox.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 4912433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_bpf.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 61161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_firmware.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7113 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_module.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/test_user_copy.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_bm.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_fsm.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/ts_kmp.ko 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/ 2024-05-27T06:31:40.347 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/lib/xz/xz_dec_test.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/ 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/mm/hwpoison-inject.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/ 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/6lowpan.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_dest.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_fragment.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_hop.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_mobility.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_routing.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/6lowpan/nhc_udp.ko 2024-05-27T06:31:40.348 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/ 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/garp.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/mrp.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/p8022.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/psnap.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13529 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/802/stp.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/ 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 106697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/8021q/8021q.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/ 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 236769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_fd.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_rdma.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/9p/9pnet_virtio.ko 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/ 2024-05-27T06:31:40.349 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 879593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/batman-adv/batman-adv.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/ 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/br_netfilter.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1031177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/bridge.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10329 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-27T06:31:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9561 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/ebtables.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-27T06:31:40.351 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/ 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1243369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ceph/libceph.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/ 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 153793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/core/pktgen.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/ 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 244825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_diag.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 71025 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv4.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 74985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dccp/dccp_ipv6.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/ 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 507649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/dsa_core.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_brcm.ko 2024-05-27T06:31:40.352 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_dsa.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_none.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/dsa/tag_trailer.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/ 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 128577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/hsr/hsr.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 357369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 59385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ah4.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/esp4.ko 2024-05-27T06:31:40.353 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/fou.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/gre.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/inet_diag.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 75481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_gre.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 77713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_tunnel.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ip_vti.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipcomp.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/ipip.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-27T06:31:40.354 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50161 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ip_tables.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9697 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-27T06:31:40.355 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_bic.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_dctcp.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_diag.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_highspeed.ko 2024-05-27T06:31:40.356 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_htcp.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11657 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_hybla.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_illinois.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_lp.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_scalable.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_vegas.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_veno.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_westwood.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tcp_yeah.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/tunnel4.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_diag.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 54425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/udp_tunnel.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-27T06:31:40.357 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ah6.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 47425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/esp6.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17105 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/fou6.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 103169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_gre.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 118873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_tunnel.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 71081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ip6_vti.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/ipcomp6.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/mip6.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-27T06:31:40.358 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11521 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-27T06:31:40.359 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 29809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 94081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/sit.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 30289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/tunnel6.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/ 2024-05-27T06:31:40.360 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 106361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/key/af_key.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/ 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 153209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_core.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_eth.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ip6.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_netlink.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 66233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/l2tp/l2tp_ppp.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/ 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 63321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/lapb/lapb.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/ 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc.ko 2024-05-27T06:31:40.361 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 332817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/llc/llc2.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/ 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 388817 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mac802154/mac802154.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/ 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/mpls/mpls_gso.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 147729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 124033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-27T06:31:40.362 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 124337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 126673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 126465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 137825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 72345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 133673 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 139065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 138257 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 136169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 139889 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 534033 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14193 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-27T06:31:40.363 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20945 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24569 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12641 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15969 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 55209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conncount.ko 2024-05-27T06:31:40.364 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 496713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 134361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24505 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 140793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 79865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12553 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 117385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table.ko 2024-05-27T06:31:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 46097 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_log_syslog.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 118337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14177 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13337 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_irc.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_sip.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 811705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nf_tables.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53041 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-05-27T06:31:40.366 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_log.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 65137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_chain_nat.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_compat.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 50649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_ct.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fib.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_hash.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_limit.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17913 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_log.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_masq.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_nat.ko 2024-05-27T06:31:40.367 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16465 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_numgen.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13937 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_osf.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_queue.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_redir.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_reject_inet.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15049 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_socket.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 24985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_synproxy.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/nft_tproxy.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 108385 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/x_tables.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_AUDIT.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-27T06:31:40.368 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 21857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_CT.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12281 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_DSCP.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11065 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HL.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15737 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_HMARK.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45209 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15929 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LED.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13009 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_LOG.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13369 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NETMAP.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10849 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFLOG.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15353 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_RATEEST.ko 2024-05-27T06:31:40.369 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16681 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_SECMARK.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18977 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11761 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20689 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TEE.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TPROXY.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_TRACE.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_addrtype.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16625 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_bpf.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cgroup.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cluster.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_comment.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11953 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connbytes.ko 2024-05-27T06:31:40.370 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlabel.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connlimit.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15833 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_connmark.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14201 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_conntrack.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_cpu.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dccp.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_devgroup.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_dscp.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12649 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ecn.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_esp.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 52809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hashlimit.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_helper.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_hl.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipcomp.ko 2024-05-27T06:31:40.371 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_iprange.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_ipvs.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19185 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_l2tp.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_length.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11513 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_limit.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mac.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_mark.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_multiport.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 16153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nat.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_nfacct.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_osf.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11489 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_owner.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_physdev.ko 2024-05-27T06:31:40.372 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_pkttype.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13545 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_policy.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_quota.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 11873 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_rateest.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9265 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_realm.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_recent.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_sctp.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36481 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_set.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 19769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_socket.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_state.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10841 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_statistic.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10017 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_string.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpmss.ko 2024-05-27T06:31:40.373 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18441 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_tcpudp.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12433 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_time.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netfilter/xt_u32.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/ 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 17225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/netlink/netlink_diag.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/ 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 14241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/nsh/nsh.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/ 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 488457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/openvswitch.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12001 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-geneve.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 10865 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-gre.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/openvswitch/vport-vxlan.ko 2024-05-27T06:31:40.374 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/ 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15921 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/packet/af_packet_diag.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/ 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 141993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/phonet.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 68801 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/phonet/pn_pep.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/ 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 318537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 202249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_rdma.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 85073 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rds/rds_tcp.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/ 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1028129 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/rxrpc/rxrpc.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/ 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38993 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_csum.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 28713 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_gact.ko 2024-05-27T06:31:40.375 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45401 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_mirred.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 27585 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_nat.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 53457 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_pedit.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38633 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_police.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23961 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_simple.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43409 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_skbedit.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 45665 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/act_vlan.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32121 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_basic.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 41449 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_bpf.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23217 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_cgroup.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 37297 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_flow.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39593 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_fw.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 49313 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_route.ko 2024-05-27T06:31:40.376 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 79137 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/cls_u32.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8705 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_cmp.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 15377 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_ipset.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 48777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_meta.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9233 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_nbyte.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13361 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_text.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 8305 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/em_u32.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_choke.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 25089 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_codel.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43473 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_drr.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 43145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 39169 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_fq_codel.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 42881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_gred.ko 2024-05-27T06:31:40.377 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 62809 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hfsc.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26577 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_hhf.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 81825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_htb.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33537 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_ingress.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 36617 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12289 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_mqprio_lib.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33057 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_multiq.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 44785 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_netem.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 32825 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_pie.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 13497 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_plug.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 31793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_prio.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 60241 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_qfq.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 40881 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_red.ko 2024-05-27T06:31:40.378 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 33425 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfb.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 38321 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_sfq.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 35769 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_tbf.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 34153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sched/sch_teql.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/ 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1187153 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23905 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sctp/sctp_diag.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/ 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/ 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 432225 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70081 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1940753 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/sunrpc.ko 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/ 2024-05-27T06:31:40.379 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 889729 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/ 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 1122273 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12145 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/tipc/tipc_diag.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/ 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 18793 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/unix/unix_diag.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/ 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 105601 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 90249 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 123745 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 12897 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 20393 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/ 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 165777 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/x25/x25.ko 2024-05-27T06:31:40.380 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/ 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 26721 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_algo.ko 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23345 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 131857 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/kernel/net/xfrm/xfrm_user.ko 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 646387 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 630543 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.alias.bin 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 7706 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 23609 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.alias.bin 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 9941 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.bin 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 70990 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.builtin.modinfo 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 146985 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 232892 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.dep.bin 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 210 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.devname 2024-05-27T06:31:40.381 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 72627 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.order 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 879 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.softdep 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 258450 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 313019 2024-05-23 11:07 ./lib/modules/6.9.0-gb9459a26b088/modules.symbols.bin 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/ 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/ 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/ 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:drwxr-xr-x root/root 0 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/ 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 171 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/changelog.Debian.gz 2024-05-27T06:31:40.382 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r-- root/root 691 2024-05-23 11:07 ./usr/share/doc/linux-image-6.9.0-gb9459a26b088/copyright 2024-05-27T06:31:40.387 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-gb9459a26b088 2024-05-27T06:31:40.387 INFO:teuthology.task.kernel:searching for kernel 6.9.0-gb9459a26b088 2024-05-27T06:31:40.387 DEBUG:teuthology.orchestra.run.smithi106:> egrep '(submenu|menuentry.*6.9.0-gb9459a26b088).*{' /boot/grub/grub.cfg 2024-05-27T06:31:40.393 INFO:teuthology.orchestra.run.smithi106.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:40.393 INFO:teuthology.orchestra.run.smithi106.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:40.393 INFO:teuthology.orchestra.run.smithi106.stdout: menuentry 'Ubuntu, with Linux 6.9.0-gb9459a26b088 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-gb9459a26b088-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-05-27T06:31:40.393 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-05-27T06:31:40.394 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-gb9459a26b088 2024-05-27T06:31:40.394 DEBUG:teuthology.orchestra.run.smithi106:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-05-27T06:31:40.439 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi106.front.sepia.ceph.com to complete... 2024-05-27T06:31:40.822 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub' 2024-05-27T06:31:40.823 INFO:teuthology.orchestra.run.smithi106.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-27T06:31:40.845 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-05-27T06:31:41.075 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-6.9.0-gb9459a26b088 2024-05-27T06:31:41.081 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-6.9.0-gb9459a26b088 2024-05-27T06:31:41.328 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-27T06:31:41.334 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-27T06:31:41.379 INFO:teuthology.orchestra.run.smithi106.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-27T06:31:41.384 INFO:teuthology.orchestra.run.smithi106.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-27T06:31:41.544 INFO:teuthology.orchestra.run.smithi106.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-27T06:31:41.544 INFO:teuthology.orchestra.run.smithi106.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-27T06:31:41.544 INFO:teuthology.orchestra.run.smithi106.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-27T06:31:41.554 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-05-27T06:32:10.432 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-27T06:32:10.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:32:10.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:32:12.737 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-27T06:32:12.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:32:12.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:32:28.959 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-05-27T06:32:31.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-05-27T06:32:37.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:32:37.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:32:40.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:32:40.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:32:41.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-05-27T06:32:43.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-05-27T06:32:53.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:32:53.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:32:55.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:32:55.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:33:11.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-05-27T06:33:13.758 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.106 2024-05-27T06:33:26.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:33:26.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:33:27.129 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-05-27T06:33:27.972 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:33:27.972 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.9.0-gb9459a26b088"... 2024-05-27T06:33:27.972 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-05-27T06:33:28.023 INFO:teuthology.orchestra.run.smithi081.stdout:6.9.0-gb9459a26b088 2024-05-27T06:33:28.023 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:33:28.023 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:33:28.023 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-27T06:33:28.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:33:28.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:33:28.973 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-05-27T06:33:29.024 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-27T06:33:29.024 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:33:29.124 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-05-27T06:33:29.194 DEBUG:teuthology.parallel:result is None 2024-05-27T06:33:29.630 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:33:29.630 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.9.0-gb9459a26b088"... 2024-05-27T06:33:29.630 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-05-27T06:33:29.679 INFO:teuthology.orchestra.run.smithi106.stdout:6.9.0-gb9459a26b088 2024-05-27T06:33:29.679 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-gb9459a26b088 vs 6.9.0-gb9459a26b088 2024-05-27T06:33:29.679 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-27T06:33:29.679 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-27T06:33:30.680 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-27T06:33:30.680 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-27T06:33:30.755 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-05-27T06:33:30.808 DEBUG:teuthology.parallel:result is None 2024-05-27T06:33:30.808 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-27T06:33:30.815 INFO:teuthology.task.internal:Creating test directory... 2024-05-27T06:33:30.816 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:33:30.819 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-27T06:33:30.827 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-27T06:33:30.834 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-27T06:33:30.842 INFO:teuthology.task.internal:Creating archive directory... 2024-05-27T06:33:30.842 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:33:30.871 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-27T06:33:30.899 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-27T06:33:30.906 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-27T06:33:30.906 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:33:30.935 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-27T06:33:30.958 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:33:30.969 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:33:30.972 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:33:30.982 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-27T06:33:30.985 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-27T06:33:30.992 INFO:teuthology.task.internal:Configuring sudo... 2024-05-27T06:33:30.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:33:31.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-27T06:33:31.047 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-27T06:33:31.057 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-27T06:33:31.057 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:33:31.078 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-27T06:33:31.103 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:33:31.136 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:33:31.188 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:33:31.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:33:31.259 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-27T06:33:31.268 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-27T06:33:31.320 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:33:31.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-27T06:33:31.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-05-27T06:33:31.396 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-05-27T06:33:31.505 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-27T06:33:31.513 INFO:teuthology.task.internal:Starting timer... 2024-05-27T06:33:31.513 INFO:teuthology.run_tasks:Running task pcp... 2024-05-27T06:33:31.523 INFO:teuthology.run_tasks:Running task selinux... 2024-05-27T06:33:31.532 DEBUG:teuthology.task.selinux:Excluding smithi081: OS 'ubuntu' does not support SELinux 2024-05-27T06:33:31.533 DEBUG:teuthology.task.selinux:Excluding smithi106: OS 'ubuntu' does not support SELinux 2024-05-27T06:33:31.533 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-27T06:33:31.533 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-27T06:33:31.533 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-27T06:33:31.533 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-27T06:33:31.542 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-27T06:33:31.559 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-27T06:33:31.560 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi106.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-27T06:41:08.319 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com')] 2024-05-27T06:41:08.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:41:08.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:41:08.404 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-05-27T06:41:08.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-05-27T06:41:08.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:41:08.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-27T06:41:08.620 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-05-27T06:41:08.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-05-27T06:41:08.753 INFO:teuthology.run_tasks:Running task clock... 2024-05-27T06:41:08.765 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-27T06:41:08.765 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T06:41:08.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T06:41:08.769 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-27T06:41:08.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T06:41:08.804 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Command line: ntpd -gq 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: ---------------------------------------------------- 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: ntp-4 is maintained by Network Time Foundation, 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: corporation. Support and training for ntp-4 are 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: available at https://www.nwtime.org/support 2024-05-27T06:41:08.805 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: ---------------------------------------------------- 2024-05-27T06:41:08.806 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: proto: precision = 0.042 usec (-24) 2024-05-27T06:41:08.806 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: basedate set to 2022-02-04 2024-05-27T06:41:08.806 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: gps base set to 2022-02-06 (week 2196) 2024-05-27T06:41:08.806 INFO:teuthology.orchestra.run.smithi081.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T06:41:08.807 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T06:41:08.807 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: restrict ::: KOD does nothing without LIMITED. 2024-05-27T06:41:08.807 INFO:teuthology.orchestra.run.smithi081.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen and drop on 0 v6wildcard [::]:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen normally on 2 lo 127.0.0.1:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen normally on 3 enp3s0f1 172.21.15.81:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen normally on 4 lo [::1]:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d753%5]:123 2024-05-27T06:41:08.808 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:08 ntpd[17565]: Listening on routing socket on fd #22 for interface updates 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Command line: ntpd -gq 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: ---------------------------------------------------- 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: ntp-4 is maintained by Network Time Foundation, 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: corporation. Support and training for ntp-4 are 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: available at https://www.nwtime.org/support 2024-05-27T06:41:08.832 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: ---------------------------------------------------- 2024-05-27T06:41:08.833 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: proto: precision = 0.041 usec (-24) 2024-05-27T06:41:08.833 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: basedate set to 2022-02-04 2024-05-27T06:41:08.833 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: gps base set to 2022-02-06 (week 2196) 2024-05-27T06:41:08.833 INFO:teuthology.orchestra.run.smithi106.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T06:41:08.834 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-27T06:41:08.834 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: restrict ::: KOD does nothing without LIMITED. 2024-05-27T06:41:08.834 INFO:teuthology.orchestra.run.smithi106.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-27T06:41:08.835 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen and drop on 0 v6wildcard [::]:123 2024-05-27T06:41:08.835 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-27T06:41:08.835 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen normally on 2 lo 127.0.0.1:123 2024-05-27T06:41:08.835 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen normally on 3 enp3s0f1 172.21.15.106:123 2024-05-27T06:41:08.836 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen normally on 4 lo [::1]:123 2024-05-27T06:41:08.836 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce6d%5]:123 2024-05-27T06:41:08.836 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:08 ntpd[17573]: Listening on routing socket on fd #22 for interface updates 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:09.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:09.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:09 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:09.832 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:09 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:09.832 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:09 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:10.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:10 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:11.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:11.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:11.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:11 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:12.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:12 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:13.808 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:13 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:14.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:14.835 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:14 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:15.806 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:15.807 INFO:teuthology.orchestra.run.smithi081.stdout:27 May 06:41:15 ntpd[17565]: ntpd: time slew -0.000233 s 2024-05-27T06:41:15.807 INFO:teuthology.orchestra.run.smithi081.stdout:ntpd: time slew -0.000233s 2024-05-27T06:41:15.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-27T06:41:15.807 INFO:teuthology.orchestra.run.smithi081.stderr:27 May 06:41:15 ntpd[17565]: can't open /var/log/ntpstats/loopstats.20240527: Permission denied 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:15.883 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/rawstats.20240527: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-27T06:41:16.833 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/peerstats.20240527: Permission denied 2024-05-27T06:41:16.834 INFO:teuthology.orchestra.run.smithi106.stdout:27 May 06:41:16 ntpd[17573]: ntpd: time slew -0.000940 s 2024-05-27T06:41:16.834 INFO:teuthology.orchestra.run.smithi106.stdout:ntpd: time slew -0.000940s 2024-05-27T06:41:16.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-27T06:41:16.834 INFO:teuthology.orchestra.run.smithi106.stderr:27 May 06:41:16 ntpd[17573]: can't open /var/log/ntpstats/loopstats.20240527: Permission denied 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================== 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:16.940 INFO:teuthology.orchestra.run.smithi106.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-27T06:41:16.941 INFO:teuthology.run_tasks:Running task install... 2024-05-27T06:41:16.949 DEBUG:teuthology.task.install:project ceph 2024-05-27T06:41:16.949 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}} 2024-05-27T06:41:16.950 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'} 2024-05-27T06:41:16.950 INFO:teuthology.task.install:Using flavor: default 2024-05-27T06:41:16.954 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-27T06:41:16.955 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-27T06:41:16.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-key list | grep Ceph 2024-05-27T06:41:16.955 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-key list | grep Ceph 2024-05-27T06:41:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-27T06:41:17.092 INFO:teuthology.orchestra.run.smithi106.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-27T06:41:17.111 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-27T06:41:17.111 INFO:teuthology.orchestra.run.smithi081.stdout:uid [ unknown] Ceph.com (release key) 2024-05-27T06:41:17.112 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-05-27T06:41:17.112 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-05-27T06:41:17.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:41:17.143 INFO:teuthology.orchestra.run.smithi106.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-27T06:41:17.143 INFO:teuthology.orchestra.run.smithi106.stdout:uid [ unknown] Ceph.com (release key) 2024-05-27T06:41:17.144 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-05-27T06:41:17.144 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-05-27T06:41:17.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:41:17.292 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default/ 2024-05-27T06:41:17.292 INFO:teuthology.task.install.deb:Package version is 19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:41:17.339 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default/ 2024-05-27T06:41:17.339 INFO:teuthology.task.install.deb:Package version is 19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:41:17.407 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:41:17.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-27T06:41:17.426 DEBUG:teuthology.orchestra.run.smithi106:> sudo apt-get update 2024-05-27T06:41:17.445 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:41:17.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-27T06:41:17.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo apt-get update 2024-05-27T06:41:17.619 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-27T06:41:17.629 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-27T06:41:17.632 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-27T06:41:17.650 INFO:teuthology.orchestra.run.smithi106.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-27T06:41:17.660 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-27T06:41:17.660 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-27T06:41:17.662 INFO:teuthology.orchestra.run.smithi106.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy InRelease 2024-05-27T06:41:17.670 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-27T06:41:17.683 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-27T06:41:17.698 INFO:teuthology.orchestra.run.smithi081.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-27T06:41:17.703 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy InRelease 2024-05-27T06:41:17.705 INFO:teuthology.orchestra.run.smithi106.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-27T06:41:17.724 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-27T06:41:17.726 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-27T06:41:17.746 INFO:teuthology.orchestra.run.smithi081.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-27T06:41:17.758 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,397 B] 2024-05-27T06:41:17.767 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,397 B] 2024-05-27T06:41:17.799 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-05-27T06:41:18.209 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 29.1 kB in 1s (52.2 kB/s) 2024-05-27T06:41:18.251 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 29.1 kB in 1s (52.2 kB/s) 2024-05-27T06:41:19.204 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2024-05-27T06:41:19.219 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3872-g38f9d900-1jammy cephadm=19.0.0-3872-g38f9d900-1jammy ceph-mds=19.0.0-3872-g38f9d900-1jammy ceph-mgr=19.0.0-3872-g38f9d900-1jammy ceph-common=19.0.0-3872-g38f9d900-1jammy ceph-fuse=19.0.0-3872-g38f9d900-1jammy ceph-test=19.0.0-3872-g38f9d900-1jammy ceph-volume=19.0.0-3872-g38f9d900-1jammy radosgw=19.0.0-3872-g38f9d900-1jammy python3-rados=19.0.0-3872-g38f9d900-1jammy python3-rgw=19.0.0-3872-g38f9d900-1jammy python3-cephfs=19.0.0-3872-g38f9d900-1jammy python3-rbd=19.0.0-3872-g38f9d900-1jammy libcephfs2=19.0.0-3872-g38f9d900-1jammy libcephfs-dev=19.0.0-3872-g38f9d900-1jammy librados2=19.0.0-3872-g38f9d900-1jammy librbd1=19.0.0-3872-g38f9d900-1jammy rbd-fuse=19.0.0-3872-g38f9d900-1jammy python3-cephfs=19.0.0-3872-g38f9d900-1jammy cephfs-shell=19.0.0-3872-g38f9d900-1jammy cephfs-top=19.0.0-3872-g38f9d900-1jammy cephfs-mirror=19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:41:19.252 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-05-27T06:41:19.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3872-g38f9d900-1jammy cephadm=19.0.0-3872-g38f9d900-1jammy ceph-mds=19.0.0-3872-g38f9d900-1jammy ceph-mgr=19.0.0-3872-g38f9d900-1jammy ceph-common=19.0.0-3872-g38f9d900-1jammy ceph-fuse=19.0.0-3872-g38f9d900-1jammy ceph-test=19.0.0-3872-g38f9d900-1jammy ceph-volume=19.0.0-3872-g38f9d900-1jammy radosgw=19.0.0-3872-g38f9d900-1jammy python3-rados=19.0.0-3872-g38f9d900-1jammy python3-rgw=19.0.0-3872-g38f9d900-1jammy python3-cephfs=19.0.0-3872-g38f9d900-1jammy python3-rbd=19.0.0-3872-g38f9d900-1jammy libcephfs2=19.0.0-3872-g38f9d900-1jammy libcephfs-dev=19.0.0-3872-g38f9d900-1jammy librados2=19.0.0-3872-g38f9d900-1jammy librbd1=19.0.0-3872-g38f9d900-1jammy rbd-fuse=19.0.0-3872-g38f9d900-1jammy python3-cephfs=19.0.0-3872-g38f9d900-1jammy cephfs-shell=19.0.0-3872-g38f9d900-1jammy cephfs-top=19.0.0-3872-g38f9d900-1jammy cephfs-mirror=19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:41:19.294 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2024-05-27T06:41:19.341 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-05-27T06:41:19.469 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2024-05-27T06:41:19.470 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2024-05-27T06:41:19.515 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2024-05-27T06:41:19.516 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2024-05-27T06:41:19.645 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T06:41:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T06:41:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T06:41:19.645 INFO:teuthology.orchestra.run.smithi106.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T06:41:19.646 INFO:teuthology.orchestra.run.smithi106.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T06:41:19.648 INFO:teuthology.orchestra.run.smithi106.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed: 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-27T06:41:19.649 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-05-27T06:41:19.650 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-05-27T06:41:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-05-27T06:41:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-05-27T06:41:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-05-27T06:41:19.651 INFO:teuthology.orchestra.run.smithi106.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-05-27T06:41:19.651 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages: 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout:Recommended packages: 2024-05-27T06:41:19.652 INFO:teuthology.orchestra.run.smithi106.stdout: btrfs-tools 2024-05-27T06:41:19.693 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T06:41:19.694 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T06:41:19.694 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T06:41:19.694 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T06:41:19.694 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T06:41:19.694 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T06:41:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T06:41:19.695 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2024-05-27T06:41:19.695 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-27T06:41:19.695 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-27T06:41:19.696 INFO:teuthology.orchestra.run.smithi081.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T06:41:19.696 INFO:teuthology.orchestra.run.smithi081.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-05-27T06:41:19.697 INFO:teuthology.orchestra.run.smithi081.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-05-27T06:41:19.698 INFO:teuthology.orchestra.run.smithi081.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout:Recommended packages: 2024-05-27T06:41:19.699 INFO:teuthology.orchestra.run.smithi081.stdout: btrfs-tools 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed: 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-05-27T06:41:19.807 INFO:teuthology.orchestra.run.smithi106.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T06:41:19.808 INFO:teuthology.orchestra.run.smithi106.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-05-27T06:41:19.808 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-05-27T06:41:19.809 INFO:teuthology.orchestra.run.smithi106.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-05-27T06:41:19.809 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-05-27T06:41:19.809 INFO:teuthology.orchestra.run.smithi106.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-27T06:41:19.810 INFO:teuthology.orchestra.run.smithi106.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-27T06:41:19.811 INFO:teuthology.orchestra.run.smithi106.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-27T06:41:19.811 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-27T06:41:19.811 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-05-27T06:41:19.811 INFO:teuthology.orchestra.run.smithi106.stdout: socat unzip xmlstarlet zip 2024-05-27T06:41:19.857 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2024-05-27T06:41:19.857 INFO:teuthology.orchestra.run.smithi081.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-27T06:41:19.857 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-27T06:41:19.857 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-05-27T06:41:19.857 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-27T06:41:19.859 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-05-27T06:41:19.860 INFO:teuthology.orchestra.run.smithi081.stdout: socat unzip xmlstarlet zip 2024-05-27T06:41:19.880 INFO:teuthology.orchestra.run.smithi106.stdout:0 upgraded, 124 newly installed, 0 to remove and 310 not upgraded. 2024-05-27T06:41:19.880 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 186 MB of archives. 2024-05-27T06:41:19.880 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 952 MB of additional disk space will be used. 2024-05-27T06:41:19.881 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-05-27T06:41:19.901 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-27T06:41:19.912 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-05-27T06:41:19.933 INFO:teuthology.orchestra.run.smithi081.stdout:0 upgraded, 124 newly installed, 0 to remove and 310 not upgraded. 2024-05-27T06:41:19.933 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 186 MB of archives. 2024-05-27T06:41:19.933 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 952 MB of additional disk space will be used. 2024-05-27T06:41:19.933 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-05-27T06:41:19.955 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-27T06:41:19.959 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-27T06:41:19.966 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-05-27T06:41:19.983 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-27T06:41:19.985 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-27T06:41:19.999 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-27T06:41:20.018 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-27T06:41:20.032 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3872-g38f9d900-1jammy [3,527 kB] 2024-05-27T06:41:20.041 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-27T06:41:20.043 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-27T06:41:20.044 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-27T06:41:20.048 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-27T06:41:20.051 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-27T06:41:20.054 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-27T06:41:20.054 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-27T06:41:20.060 INFO:teuthology.orchestra.run.smithi106.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-27T06:41:20.062 INFO:teuthology.orchestra.run.smithi106.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-27T06:41:20.064 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-27T06:41:20.064 INFO:teuthology.orchestra.run.smithi106.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-27T06:41:20.065 INFO:teuthology.orchestra.run.smithi106.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-27T06:41:20.065 INFO:teuthology.orchestra.run.smithi106.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-27T06:41:20.082 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3872-g38f9d900-1jammy [3,527 kB] 2024-05-27T06:41:20.084 INFO:teuthology.orchestra.run.smithi106.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-27T06:41:20.086 INFO:teuthology.orchestra.run.smithi106.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-27T06:41:20.087 INFO:teuthology.orchestra.run.smithi106.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-27T06:41:20.088 INFO:teuthology.orchestra.run.smithi106.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-27T06:41:20.088 INFO:teuthology.orchestra.run.smithi106.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-27T06:41:20.103 INFO:teuthology.orchestra.run.smithi106.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-27T06:41:20.104 INFO:teuthology.orchestra.run.smithi106.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-27T06:41:20.104 INFO:teuthology.orchestra.run.smithi106.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-27T06:41:20.105 INFO:teuthology.orchestra.run.smithi106.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-27T06:41:20.105 INFO:teuthology.orchestra.run.smithi106.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-27T06:41:20.112 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-27T06:41:20.114 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-27T06:41:20.122 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-27T06:41:20.123 INFO:teuthology.orchestra.run.smithi106.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-27T06:41:20.123 INFO:teuthology.orchestra.run.smithi106.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-27T06:41:20.125 INFO:teuthology.orchestra.run.smithi106.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-27T06:41:20.126 INFO:teuthology.orchestra.run.smithi106.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-27T06:41:20.126 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-27T06:41:20.127 INFO:teuthology.orchestra.run.smithi106.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-27T06:41:20.127 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-27T06:41:20.133 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-27T06:41:20.135 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-27T06:41:20.137 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-27T06:41:20.138 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-27T06:41:20.138 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-27T06:41:20.142 INFO:teuthology.orchestra.run.smithi106.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-27T06:41:20.143 INFO:teuthology.orchestra.run.smithi106.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-27T06:41:20.143 INFO:teuthology.orchestra.run.smithi106.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-27T06:41:20.144 INFO:teuthology.orchestra.run.smithi106.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-27T06:41:20.145 INFO:teuthology.orchestra.run.smithi106.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-27T06:41:20.157 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-27T06:41:20.159 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-27T06:41:20.160 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-27T06:41:20.161 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-27T06:41:20.161 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-27T06:41:20.162 INFO:teuthology.orchestra.run.smithi106.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-27T06:41:20.163 INFO:teuthology.orchestra.run.smithi106.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-27T06:41:20.167 INFO:teuthology.orchestra.run.smithi106.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-27T06:41:20.167 INFO:teuthology.orchestra.run.smithi106.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-27T06:41:20.168 INFO:teuthology.orchestra.run.smithi106.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-27T06:41:20.178 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-27T06:41:20.179 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-27T06:41:20.179 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-27T06:41:20.179 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-27T06:41:20.179 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-27T06:41:20.182 INFO:teuthology.orchestra.run.smithi106.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-27T06:41:20.183 INFO:teuthology.orchestra.run.smithi106.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-27T06:41:20.185 INFO:teuthology.orchestra.run.smithi106.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-27T06:41:20.185 INFO:teuthology.orchestra.run.smithi106.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-27T06:41:20.187 INFO:teuthology.orchestra.run.smithi106.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-27T06:41:20.199 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-27T06:41:20.199 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-27T06:41:20.201 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-27T06:41:20.201 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-27T06:41:20.202 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-27T06:41:20.202 INFO:teuthology.orchestra.run.smithi106.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-27T06:41:20.204 INFO:teuthology.orchestra.run.smithi106.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-27T06:41:20.205 INFO:teuthology.orchestra.run.smithi106.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-27T06:41:20.206 INFO:teuthology.orchestra.run.smithi106.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-27T06:41:20.207 INFO:teuthology.orchestra.run.smithi106.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-27T06:41:20.241 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-27T06:41:20.242 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-27T06:41:20.242 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-27T06:41:20.243 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-27T06:41:20.244 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-27T06:41:20.266 INFO:teuthology.orchestra.run.smithi106.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-27T06:41:20.266 INFO:teuthology.orchestra.run.smithi106.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-27T06:41:20.268 INFO:teuthology.orchestra.run.smithi106.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-27T06:41:20.268 INFO:teuthology.orchestra.run.smithi106.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-27T06:41:20.269 INFO:teuthology.orchestra.run.smithi106.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-27T06:41:20.277 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-27T06:41:20.277 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-27T06:41:20.282 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-27T06:41:20.282 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-27T06:41:20.283 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-27T06:41:20.295 INFO:teuthology.orchestra.run.smithi106.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-27T06:41:20.297 INFO:teuthology.orchestra.run.smithi106.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-27T06:41:20.298 INFO:teuthology.orchestra.run.smithi106.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-27T06:41:20.319 INFO:teuthology.orchestra.run.smithi081.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-27T06:41:20.320 INFO:teuthology.orchestra.run.smithi081.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-27T06:41:20.321 INFO:teuthology.orchestra.run.smithi081.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-05-27T06:41:20.322 INFO:teuthology.orchestra.run.smithi081.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-27T06:41:20.323 INFO:teuthology.orchestra.run.smithi081.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-27T06:41:20.327 INFO:teuthology.orchestra.run.smithi106.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-27T06:41:20.327 INFO:teuthology.orchestra.run.smithi106.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-27T06:41:20.328 INFO:teuthology.orchestra.run.smithi106.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-27T06:41:20.329 INFO:teuthology.orchestra.run.smithi106.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-27T06:41:20.329 INFO:teuthology.orchestra.run.smithi106.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-27T06:41:20.330 INFO:teuthology.orchestra.run.smithi106.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-27T06:41:20.339 INFO:teuthology.orchestra.run.smithi081.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-27T06:41:20.341 INFO:teuthology.orchestra.run.smithi081.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-27T06:41:20.342 INFO:teuthology.orchestra.run.smithi081.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-27T06:41:20.342 INFO:teuthology.orchestra.run.smithi081.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-27T06:41:20.345 INFO:teuthology.orchestra.run.smithi081.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-27T06:41:20.354 INFO:teuthology.orchestra.run.smithi106.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-27T06:41:20.354 INFO:teuthology.orchestra.run.smithi106.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-27T06:41:20.365 INFO:teuthology.orchestra.run.smithi106.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-27T06:41:20.392 INFO:teuthology.orchestra.run.smithi106.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-27T06:41:20.394 INFO:teuthology.orchestra.run.smithi106.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-27T06:41:20.395 INFO:teuthology.orchestra.run.smithi106.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-27T06:41:20.401 INFO:teuthology.orchestra.run.smithi106.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-27T06:41:20.406 INFO:teuthology.orchestra.run.smithi106.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-05-27T06:41:20.406 INFO:teuthology.orchestra.run.smithi106.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-27T06:41:20.408 INFO:teuthology.orchestra.run.smithi106.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-27T06:41:20.415 INFO:teuthology.orchestra.run.smithi081.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-27T06:41:20.416 INFO:teuthology.orchestra.run.smithi081.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-27T06:41:20.417 INFO:teuthology.orchestra.run.smithi081.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-27T06:41:20.417 INFO:teuthology.orchestra.run.smithi081.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-27T06:41:20.418 INFO:teuthology.orchestra.run.smithi081.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-27T06:41:20.433 INFO:teuthology.orchestra.run.smithi106.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-27T06:41:20.437 INFO:teuthology.orchestra.run.smithi081.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-27T06:41:20.440 INFO:teuthology.orchestra.run.smithi081.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-27T06:41:20.441 INFO:teuthology.orchestra.run.smithi106.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-27T06:41:20.441 INFO:teuthology.orchestra.run.smithi081.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-27T06:41:20.441 INFO:teuthology.orchestra.run.smithi106.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-27T06:41:20.460 INFO:teuthology.orchestra.run.smithi106.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3872-g38f9d900-1jammy [3,286 kB] 2024-05-27T06:41:20.466 INFO:teuthology.orchestra.run.smithi081.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-27T06:41:20.466 INFO:teuthology.orchestra.run.smithi081.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-27T06:41:20.467 INFO:teuthology.orchestra.run.smithi081.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-27T06:41:20.467 INFO:teuthology.orchestra.run.smithi081.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-27T06:41:20.468 INFO:teuthology.orchestra.run.smithi081.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-27T06:41:20.468 INFO:teuthology.orchestra.run.smithi106.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-27T06:41:20.469 INFO:teuthology.orchestra.run.smithi081.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-27T06:41:20.469 INFO:teuthology.orchestra.run.smithi106.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-27T06:41:20.469 INFO:teuthology.orchestra.run.smithi106.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-27T06:41:20.470 INFO:teuthology.orchestra.run.smithi106.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-27T06:41:20.471 INFO:teuthology.orchestra.run.smithi106.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-27T06:41:20.472 INFO:teuthology.orchestra.run.smithi106.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-27T06:41:20.479 INFO:teuthology.orchestra.run.smithi106.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-27T06:41:20.485 INFO:teuthology.orchestra.run.smithi081.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-27T06:41:20.486 INFO:teuthology.orchestra.run.smithi081.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-27T06:41:20.490 INFO:teuthology.orchestra.run.smithi081.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-27T06:41:20.490 INFO:teuthology.orchestra.run.smithi106.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-27T06:41:20.490 INFO:teuthology.orchestra.run.smithi106.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-27T06:41:20.492 INFO:teuthology.orchestra.run.smithi106.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-27T06:41:20.506 INFO:teuthology.orchestra.run.smithi081.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-27T06:41:20.508 INFO:teuthology.orchestra.run.smithi081.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-27T06:41:20.508 INFO:teuthology.orchestra.run.smithi081.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-27T06:41:20.512 INFO:teuthology.orchestra.run.smithi081.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-27T06:41:20.514 INFO:teuthology.orchestra.run.smithi106.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-27T06:41:20.515 INFO:teuthology.orchestra.run.smithi081.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-05-27T06:41:20.515 INFO:teuthology.orchestra.run.smithi081.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-27T06:41:20.516 INFO:teuthology.orchestra.run.smithi081.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-27T06:41:20.536 INFO:teuthology.orchestra.run.smithi081.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-27T06:41:20.541 INFO:teuthology.orchestra.run.smithi081.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-27T06:41:20.542 INFO:teuthology.orchestra.run.smithi081.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-27T06:41:20.557 INFO:teuthology.orchestra.run.smithi081.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-27T06:41:20.557 INFO:teuthology.orchestra.run.smithi081.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-27T06:41:20.558 INFO:teuthology.orchestra.run.smithi081.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-27T06:41:20.558 INFO:teuthology.orchestra.run.smithi081.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-27T06:41:20.560 INFO:teuthology.orchestra.run.smithi081.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-27T06:41:20.561 INFO:teuthology.orchestra.run.smithi081.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-27T06:41:20.568 INFO:teuthology.orchestra.run.smithi081.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-27T06:41:20.574 INFO:teuthology.orchestra.run.smithi081.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3872-g38f9d900-1jammy [3,286 kB] 2024-05-27T06:41:20.578 INFO:teuthology.orchestra.run.smithi081.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-27T06:41:20.578 INFO:teuthology.orchestra.run.smithi081.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-27T06:41:20.580 INFO:teuthology.orchestra.run.smithi081.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-27T06:41:20.598 INFO:teuthology.orchestra.run.smithi081.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-27T06:41:20.646 INFO:teuthology.orchestra.run.smithi106.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3872-g38f9d900-1jammy [727 kB] 2024-05-27T06:41:20.668 INFO:teuthology.orchestra.run.smithi106.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3872-g38f9d900-1jammy [356 kB] 2024-05-27T06:41:20.707 INFO:teuthology.orchestra.run.smithi106.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3872-g38f9d900-1jammy [32.8 kB] 2024-05-27T06:41:20.708 INFO:teuthology.orchestra.run.smithi106.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3872-g38f9d900-1jammy [180 kB] 2024-05-27T06:41:20.714 INFO:teuthology.orchestra.run.smithi106.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3872-g38f9d900-1jammy [67.5 kB] 2024-05-27T06:41:20.716 INFO:teuthology.orchestra.run.smithi106.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3872-g38f9d900-1jammy [334 kB] 2024-05-27T06:41:20.727 INFO:teuthology.orchestra.run.smithi106.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3872-g38f9d900-1jammy [8,752 kB] 2024-05-27T06:41:20.797 INFO:teuthology.orchestra.run.smithi081.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3872-g38f9d900-1jammy [727 kB] 2024-05-27T06:41:20.820 INFO:teuthology.orchestra.run.smithi081.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3872-g38f9d900-1jammy [356 kB] 2024-05-27T06:41:20.866 INFO:teuthology.orchestra.run.smithi081.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3872-g38f9d900-1jammy [32.8 kB] 2024-05-27T06:41:20.867 INFO:teuthology.orchestra.run.smithi081.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3872-g38f9d900-1jammy [180 kB] 2024-05-27T06:41:20.873 INFO:teuthology.orchestra.run.smithi081.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3872-g38f9d900-1jammy [67.5 kB] 2024-05-27T06:41:20.875 INFO:teuthology.orchestra.run.smithi081.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3872-g38f9d900-1jammy [334 kB] 2024-05-27T06:41:20.923 INFO:teuthology.orchestra.run.smithi081.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3872-g38f9d900-1jammy [8,752 kB] 2024-05-27T06:41:21.283 INFO:teuthology.orchestra.run.smithi106.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3872-g38f9d900-1jammy [112 kB] 2024-05-27T06:41:21.284 INFO:teuthology.orchestra.run.smithi106.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3872-g38f9d900-1jammy [471 kB] 2024-05-27T06:41:21.299 INFO:teuthology.orchestra.run.smithi106.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3872-g38f9d900-1jammy [28.8 MB] 2024-05-27T06:41:21.625 INFO:teuthology.orchestra.run.smithi081.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3872-g38f9d900-1jammy [112 kB] 2024-05-27T06:41:21.626 INFO:teuthology.orchestra.run.smithi081.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3872-g38f9d900-1jammy [471 kB] 2024-05-27T06:41:21.674 INFO:teuthology.orchestra.run.smithi081.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3872-g38f9d900-1jammy [28.8 MB] 2024-05-27T06:41:23.231 INFO:teuthology.orchestra.run.smithi106.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3872-g38f9d900-1jammy [4,915 kB] 2024-05-27T06:41:23.433 INFO:teuthology.orchestra.run.smithi081.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3872-g38f9d900-1jammy [4,915 kB] 2024-05-27T06:41:23.541 INFO:teuthology.orchestra.run.smithi106.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3872-g38f9d900-1jammy [245 kB] 2024-05-27T06:41:23.549 INFO:teuthology.orchestra.run.smithi106.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3872-g38f9d900-1jammy [125 kB] 2024-05-27T06:41:23.552 INFO:teuthology.orchestra.run.smithi106.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3872-g38f9d900-1jammy [1,541 kB] 2024-05-27T06:41:23.643 INFO:teuthology.orchestra.run.smithi106.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3872-g38f9d900-1jammy [6,149 kB] 2024-05-27T06:41:23.744 INFO:teuthology.orchestra.run.smithi081.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3872-g38f9d900-1jammy [245 kB] 2024-05-27T06:41:23.750 INFO:teuthology.orchestra.run.smithi081.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3872-g38f9d900-1jammy [125 kB] 2024-05-27T06:41:23.754 INFO:teuthology.orchestra.run.smithi081.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3872-g38f9d900-1jammy [1,541 kB] 2024-05-27T06:41:23.862 INFO:teuthology.orchestra.run.smithi081.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3872-g38f9d900-1jammy [6,149 kB] 2024-05-27T06:41:24.047 INFO:teuthology.orchestra.run.smithi106.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3872-g38f9d900-1jammy [22.5 MB] 2024-05-27T06:41:24.261 INFO:teuthology.orchestra.run.smithi081.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3872-g38f9d900-1jammy [22.5 MB] 2024-05-27T06:41:25.512 INFO:teuthology.orchestra.run.smithi106.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3872-g38f9d900-1jammy [14.0 kB] 2024-05-27T06:41:25.513 INFO:teuthology.orchestra.run.smithi106.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3872-g38f9d900-1jammy [857 kB] 2024-05-27T06:41:25.573 INFO:teuthology.orchestra.run.smithi106.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3872-g38f9d900-1jammy [2,378 kB] 2024-05-27T06:41:25.699 INFO:teuthology.orchestra.run.smithi106.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3872-g38f9d900-1jammy [792 kB] 2024-05-27T06:41:25.768 INFO:teuthology.orchestra.run.smithi106.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3872-g38f9d900-1jammy [149 kB] 2024-05-27T06:41:25.774 INFO:teuthology.orchestra.run.smithi106.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3872-g38f9d900-1jammy [2,226 kB] 2024-05-27T06:41:25.818 INFO:teuthology.orchestra.run.smithi081.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3872-g38f9d900-1jammy [14.0 kB] 2024-05-27T06:41:25.819 INFO:teuthology.orchestra.run.smithi081.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3872-g38f9d900-1jammy [857 kB] 2024-05-27T06:41:25.839 INFO:teuthology.orchestra.run.smithi081.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3872-g38f9d900-1jammy [2,378 kB] 2024-05-27T06:41:25.891 INFO:teuthology.orchestra.run.smithi106.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3872-g38f9d900-1jammy [8,625 kB] 2024-05-27T06:41:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3872-g38f9d900-1jammy [792 kB] 2024-05-27T06:41:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3872-g38f9d900-1jammy [149 kB] 2024-05-27T06:41:26.093 INFO:teuthology.orchestra.run.smithi081.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3872-g38f9d900-1jammy [2,226 kB] 2024-05-27T06:41:26.217 INFO:teuthology.orchestra.run.smithi081.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3872-g38f9d900-1jammy [8,625 kB] 2024-05-27T06:41:26.445 INFO:teuthology.orchestra.run.smithi106.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3872-g38f9d900-1jammy [14.2 kB] 2024-05-27T06:41:26.446 INFO:teuthology.orchestra.run.smithi106.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3872-g38f9d900-1jammy [48.1 MB] 2024-05-27T06:41:26.868 INFO:teuthology.orchestra.run.smithi081.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3872-g38f9d900-1jammy [14.2 kB] 2024-05-27T06:41:26.868 INFO:teuthology.orchestra.run.smithi081.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3872-g38f9d900-1jammy [48.1 MB] 2024-05-27T06:41:29.464 INFO:teuthology.orchestra.run.smithi106.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3872-g38f9d900-1jammy [124 kB] 2024-05-27T06:41:29.465 INFO:teuthology.orchestra.run.smithi106.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3872-g38f9d900-1jammy [237 kB] 2024-05-27T06:41:29.467 INFO:teuthology.orchestra.run.smithi106.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3872-g38f9d900-1jammy [33.9 kB] 2024-05-27T06:41:29.468 INFO:teuthology.orchestra.run.smithi106.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3872-g38f9d900-1jammy [23.4 kB] 2024-05-27T06:41:29.468 INFO:teuthology.orchestra.run.smithi106.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3872-g38f9d900-1jammy [38.9 kB] 2024-05-27T06:41:29.469 INFO:teuthology.orchestra.run.smithi106.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3872-g38f9d900-1jammy [17.4 MB] 2024-05-27T06:41:30.169 INFO:teuthology.orchestra.run.smithi081.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3872-g38f9d900-1jammy [124 kB] 2024-05-27T06:41:30.192 INFO:teuthology.orchestra.run.smithi081.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3872-g38f9d900-1jammy [237 kB] 2024-05-27T06:41:30.199 INFO:teuthology.orchestra.run.smithi081.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3872-g38f9d900-1jammy [33.9 kB] 2024-05-27T06:41:30.201 INFO:teuthology.orchestra.run.smithi081.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3872-g38f9d900-1jammy [23.4 kB] 2024-05-27T06:41:30.202 INFO:teuthology.orchestra.run.smithi081.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3872-g38f9d900-1jammy [38.9 kB] 2024-05-27T06:41:30.203 INFO:teuthology.orchestra.run.smithi081.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3872-g38f9d900-1jammy [17.4 MB] 2024-05-27T06:41:30.562 INFO:teuthology.orchestra.run.smithi106.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3872-g38f9d900-1jammy [92.6 kB] 2024-05-27T06:41:30.961 INFO:teuthology.orchestra.run.smithi081.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240521.063558-debug/38f9d900c83c63799fbdbe61acc9a11b0d3554a6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3872-g38f9d900-1jammy [92.6 kB] 2024-05-27T06:41:31.197 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 186 MB in 11s (17.3 MB/s) 2024-05-27T06:41:31.347 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyperclip. 2024-05-27T06:41:31.413 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132924 files and directories currently installed.) 2024-05-27T06:41:31.416 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-05-27T06:41:31.463 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-05-27T06:41:31.601 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 186 MB in 11s (16.8 MB/s) 2024-05-27T06:41:31.731 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyperclip. 2024-05-27T06:41:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132924 files and directories currently installed.) 2024-05-27T06:41:31.802 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-05-27T06:41:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-05-27T06:41:31.940 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-27T06:41:31.954 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-27T06:41:31.990 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T06:41:32.116 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-27T06:41:32.130 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-27T06:41:32.174 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T06:41:32.293 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cmd2. 2024-05-27T06:41:32.307 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-05-27T06:41:32.352 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-05-27T06:41:32.493 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cmd2. 2024-05-27T06:41:32.506 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-05-27T06:41:32.543 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-05-27T06:41:32.722 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-27T06:41:32.736 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-27T06:41:32.822 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T06:41:32.888 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-27T06:41:32.901 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-27T06:41:32.988 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T06:41:33.141 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-27T06:41:33.156 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-27T06:41:33.200 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T06:41:33.275 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-27T06:41:33.295 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-27T06:41:33.333 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T06:41:33.520 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-27T06:41:33.534 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-27T06:41:33.570 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T06:41:33.661 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-27T06:41:33.675 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-27T06:41:33.720 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T06:41:33.998 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-27T06:41:34.013 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:34.065 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:34.140 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-27T06:41:34.154 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:34.206 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:34.510 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-27T06:41:34.524 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:34.568 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:34.609 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-27T06:41:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:34.871 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-27T06:41:34.885 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:34.921 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:35.004 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-27T06:41:35.019 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-27T06:41:35.062 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:41:35.266 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-27T06:41:35.280 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-27T06:41:35.324 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T06:41:35.407 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-27T06:41:35.423 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-27T06:41:35.482 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T06:41:35.644 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librados2. 2024-05-27T06:41:35.658 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../010-librados2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:35.702 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librados2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:35.760 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librados2. 2024-05-27T06:41:35.775 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../010-librados2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:35.810 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librados2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:36.163 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librbd1. 2024-05-27T06:41:36.177 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../011-librbd1_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:36.221 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librbd1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:36.505 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librbd1. 2024-05-27T06:41:36.520 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../011-librbd1_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:36.563 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librbd1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:36.599 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libcephfs2. 2024-05-27T06:41:36.613 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:36.649 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libcephfs2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:36.975 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs2. 2024-05-27T06:41:36.989 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:37.033 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:37.127 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rados. 2024-05-27T06:41:37.141 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:37.185 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rados (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:37.385 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rados. 2024-05-27T06:41:37.400 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:37.435 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rados (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:37.504 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-27T06:41:37.519 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:37.563 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-ceph-argparse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:37.730 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-27T06:41:37.745 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:37.788 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-argparse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:37.790 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cephfs. 2024-05-27T06:41:37.805 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:37.840 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cephfs (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:38.024 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cephfs. 2024-05-27T06:41:38.039 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:38.082 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cephfs (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:38.118 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-27T06:41:38.132 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:38.185 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:38.402 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-27T06:41:38.416 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:38.459 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:38.504 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-prettytable. 2024-05-27T06:41:38.519 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-05-27T06:41:38.562 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-27T06:41:38.796 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-prettytable. 2024-05-27T06:41:38.806 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rbd. 2024-05-27T06:41:38.811 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-05-27T06:41:38.821 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:38.854 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-27T06:41:38.902 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rbd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:39.107 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rbd. 2024-05-27T06:41:39.121 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:39.173 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rbd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:39.253 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-27T06:41:39.267 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-27T06:41:39.310 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T06:41:39.444 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-27T06:41:39.459 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-27T06:41:39.501 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T06:41:39.537 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-27T06:41:39.552 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-27T06:41:39.595 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T06:41:39.788 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-27T06:41:39.801 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-27T06:41:39.845 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T06:41:39.932 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-27T06:41:39.947 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-27T06:41:39.990 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T06:41:40.173 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-27T06:41:40.187 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-27T06:41:40.232 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T06:41:40.335 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua5.1. 2024-05-27T06:41:40.350 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-27T06:41:40.394 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-27T06:41:40.546 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua5.1. 2024-05-27T06:41:40.560 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-27T06:41:40.604 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-27T06:41:40.688 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-any. 2024-05-27T06:41:40.703 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-05-27T06:41:40.738 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-27T06:41:40.915 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-any. 2024-05-27T06:41:40.930 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-05-27T06:41:40.983 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package zip. 2024-05-27T06:41:40.996 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-05-27T06:41:40.998 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-27T06:41:41.041 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking zip (3.0-12build2) ... 2024-05-27T06:41:41.201 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package zip. 2024-05-27T06:41:41.216 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-05-27T06:41:41.251 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking zip (3.0-12build2) ... 2024-05-27T06:41:41.352 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package unzip. 2024-05-27T06:41:41.366 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-27T06:41:41.410 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-27T06:41:41.546 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package unzip. 2024-05-27T06:41:41.560 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-27T06:41:41.596 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-27T06:41:41.773 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package luarocks. 2024-05-27T06:41:41.787 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-27T06:41:41.839 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-27T06:41:42.000 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package luarocks. 2024-05-27T06:41:42.015 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-27T06:41:42.058 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-27T06:41:42.181 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package librgw2. 2024-05-27T06:41:42.194 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../027-librgw2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:42.238 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking librgw2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:42.450 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package librgw2. 2024-05-27T06:41:42.463 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../027-librgw2_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:42.507 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking librgw2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:42.900 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rgw. 2024-05-27T06:41:42.914 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:42.949 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:43.152 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rgw. 2024-05-27T06:41:43.167 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:43.210 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:43.269 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-27T06:41:43.284 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-27T06:41:43.328 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T06:41:43.547 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-27T06:41:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-27T06:41:43.605 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T06:41:43.664 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libradosstriper1. 2024-05-27T06:41:43.678 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:43.722 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libradosstriper1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:43.892 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libradosstriper1. 2024-05-27T06:41:43.906 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:43.941 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libradosstriper1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:44.142 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-common. 2024-05-27T06:41:44.156 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:44.200 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:44.336 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-common. 2024-05-27T06:41:44.351 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:44.394 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:45.549 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-base. 2024-05-27T06:41:45.564 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:45.616 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-base (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:45.775 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-base. 2024-05-27T06:41:45.790 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:45.842 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-base (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:46.186 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-27T06:41:46.200 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-27T06:41:46.270 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-27T06:41:46.573 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cheroot. 2024-05-27T06:41:46.587 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-27T06:41:46.631 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T06:41:46.696 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-27T06:41:46.710 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-27T06:41:46.754 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-27T06:41:46.951 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-27T06:41:46.965 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-27T06:41:47.009 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-27T06:41:47.057 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cheroot. 2024-05-27T06:41:47.072 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-27T06:41:47.107 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T06:41:47.385 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-27T06:41:47.399 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-27T06:41:47.460 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-27T06:41:47.529 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-27T06:41:47.543 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-27T06:41:47.587 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-27T06:41:47.747 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-27T06:41:47.761 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-27T06:41:47.805 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-27T06:41:48.108 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-27T06:41:48.122 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-27T06:41:48.123 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-27T06:41:48.138 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-27T06:41:48.166 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-27T06:41:48.182 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-27T06:41:48.468 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-tempora. 2024-05-27T06:41:48.469 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempora. 2024-05-27T06:41:48.482 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-05-27T06:41:48.483 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-05-27T06:41:48.519 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-27T06:41:48.526 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-27T06:41:48.755 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-portend. 2024-05-27T06:41:48.770 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-05-27T06:41:48.779 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-portend. 2024-05-27T06:41:48.793 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-05-27T06:41:48.813 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-27T06:41:48.837 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-27T06:41:49.132 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-27T06:41:49.147 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-27T06:41:49.183 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-27T06:41:49.190 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-27T06:41:49.198 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-27T06:41:49.266 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-27T06:41:49.477 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-27T06:41:49.491 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-27T06:41:49.535 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-27T06:41:49.569 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-27T06:41:49.584 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-27T06:41:49.627 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-27T06:41:49.880 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-natsort. 2024-05-27T06:41:49.895 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-05-27T06:41:49.938 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-27T06:41:49.981 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-natsort. 2024-05-27T06:41:49.996 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-05-27T06:41:50.039 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-27T06:41:50.233 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-logutils. 2024-05-27T06:41:50.247 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-05-27T06:41:50.291 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-27T06:41:50.384 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-logutils. 2024-05-27T06:41:50.399 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-05-27T06:41:50.443 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-27T06:41:50.594 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-27T06:41:50.609 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-27T06:41:50.652 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-27T06:41:50.737 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-27T06:41:50.752 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-27T06:41:50.804 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-27T06:41:50.947 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-mako. 2024-05-27T06:41:50.962 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-27T06:41:50.997 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T06:41:51.065 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-mako. 2024-05-27T06:41:51.080 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-27T06:41:51.131 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T06:41:51.292 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-27T06:41:51.307 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-27T06:41:51.342 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-27T06:41:51.560 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-27T06:41:51.575 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-27T06:41:51.618 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-27T06:41:51.669 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-27T06:41:51.684 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-27T06:41:51.736 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-27T06:41:52.064 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-webob. 2024-05-27T06:41:52.079 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-27T06:41:52.122 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-27T06:41:52.425 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-waitress. 2024-05-27T06:41:52.441 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-27T06:41:52.478 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T06:41:52.572 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-27T06:41:52.586 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-27T06:41:52.630 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-27T06:41:52.933 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webob. 2024-05-27T06:41:52.948 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-27T06:41:52.990 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-27T06:41:53.327 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-waitress. 2024-05-27T06:41:53.342 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-27T06:41:53.380 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T06:41:53.479 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-tempita. 2024-05-27T06:41:53.494 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-27T06:41:53.554 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T06:41:53.697 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-tempita. 2024-05-27T06:41:53.712 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-27T06:41:53.755 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T06:41:53.824 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-paste. 2024-05-27T06:41:53.838 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-27T06:41:53.874 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T06:41:54.066 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-paste. 2024-05-27T06:41:54.081 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-27T06:41:54.124 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T06:41:54.185 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-27T06:41:54.199 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-27T06:41:54.243 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T06:41:54.469 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-27T06:41:54.483 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-27T06:41:54.527 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T06:41:54.529 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-27T06:41:54.544 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-27T06:41:54.588 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-27T06:41:54.822 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-27T06:41:54.837 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-27T06:41:54.872 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-27T06:41:54.899 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-webtest. 2024-05-27T06:41:54.914 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-05-27T06:41:54.957 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-27T06:41:55.226 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-webtest. 2024-05-27T06:41:55.240 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-05-27T06:41:55.252 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pecan. 2024-05-27T06:41:55.267 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-27T06:41:55.284 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-27T06:41:55.342 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T06:41:55.595 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pecan. 2024-05-27T06:41:55.610 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-27T06:41:55.653 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T06:41:55.707 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-certifi. 2024-05-27T06:41:55.721 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-05-27T06:41:55.765 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-27T06:41:56.033 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-certifi. 2024-05-27T06:41:56.048 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-05-27T06:41:56.076 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-idna. 2024-05-27T06:41:56.091 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-27T06:41:56.091 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-27T06:41:56.134 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T06:41:56.419 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-idna. 2024-05-27T06:41:56.434 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../057-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-05-27T06:41:56.471 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-urllib3. 2024-05-27T06:41:56.477 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T06:41:56.485 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-27T06:41:56.529 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T06:41:56.799 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-urllib3. 2024-05-27T06:41:56.814 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-27T06:41:56.841 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-requests. 2024-05-27T06:41:56.855 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T06:41:56.856 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-27T06:41:56.890 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T06:41:57.151 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests. 2024-05-27T06:41:57.165 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-27T06:41:57.202 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-27T06:41:57.209 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T06:41:57.217 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-27T06:41:57.252 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T06:41:57.520 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-27T06:41:57.535 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-27T06:41:57.538 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-27T06:41:57.553 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:57.578 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T06:41:57.596 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:57.907 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-27T06:41:57.921 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:41:57.935 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-27T06:41:57.949 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:57.965 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:57.993 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:58.304 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-27T06:41:58.319 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-27T06:41:58.361 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-27T06:41:58.362 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-27T06:41:58.376 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:58.412 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:58.657 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-openssl. 2024-05-27T06:41:58.671 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-05-27T06:41:58.714 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-27T06:41:58.715 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-27T06:41:58.729 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-27T06:41:58.772 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-27T06:41:59.075 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-openssl. 2024-05-27T06:41:59.076 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr. 2024-05-27T06:41:59.090 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-05-27T06:41:59.091 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:59.126 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:59.134 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-27T06:41:59.470 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr. 2024-05-27T06:41:59.485 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:59.496 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mon. 2024-05-27T06:41:59.511 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:59.553 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:59.554 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mon (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:41:59.923 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mon. 2024-05-27T06:41:59.938 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:41:59.973 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mon (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:00.166 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-osd. 2024-05-27T06:42:00.181 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:00.224 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-osd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:00.552 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-osd. 2024-05-27T06:42:00.566 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:00.610 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-osd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:00.978 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph. 2024-05-27T06:42:00.992 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../068-ceph_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:01.027 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:01.297 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-fuse. 2024-05-27T06:42:01.311 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:01.355 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:01.413 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph. 2024-05-27T06:42:01.427 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../068-ceph_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:01.480 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:01.742 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mds. 2024-05-27T06:42:01.756 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:01.799 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-fuse. 2024-05-27T06:42:01.800 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mds (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:01.813 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:01.849 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:02.186 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cephadm. 2024-05-27T06:42:02.201 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../071-cephadm_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:02.202 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mds. 2024-05-27T06:42:02.216 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:02.244 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:02.294 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mds (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:02.514 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-27T06:42:02.529 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-27T06:42:02.564 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-27T06:42:02.696 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephadm. 2024-05-27T06:42:02.710 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../071-cephadm_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:02.755 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:03.066 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-27T06:42:03.080 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-27T06:42:03.083 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-27T06:42:03.098 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T06:42:03.124 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-27T06:42:03.133 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:03.452 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-27T06:42:03.465 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T06:42:03.510 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:04.154 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-babel. 2024-05-27T06:42:04.169 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T06:42:04.213 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:04.499 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-babel. 2024-05-27T06:42:04.507 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-jinja2. 2024-05-27T06:42:04.512 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-27T06:42:04.521 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-27T06:42:04.557 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:04.566 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T06:42:04.876 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-jinja2. 2024-05-27T06:42:04.885 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-27T06:42:04.890 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-27T06:42:04.900 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:04.934 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T06:42:04.943 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:05.313 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-27T06:42:05.313 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-27T06:42:05.327 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:05.328 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-05-27T06:42:05.364 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-27T06:42:05.371 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:05.625 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-routes. 2024-05-27T06:42:05.640 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-27T06:42:05.683 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-27T06:42:05.683 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T06:42:05.697 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-05-27T06:42:05.741 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-27T06:42:05.970 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-27T06:42:05.984 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:06.019 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:06.069 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-routes. 2024-05-27T06:42:06.083 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-27T06:42:06.118 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T06:42:06.446 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-27T06:42:06.460 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:06.513 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:07.233 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-27T06:42:07.247 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-27T06:42:07.291 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T06:42:07.745 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-joblib. 2024-05-27T06:42:07.758 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-27T06:42:07.803 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T06:42:07.844 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-27T06:42:07.858 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-27T06:42:07.902 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T06:42:08.218 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-27T06:42:08.228 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-27T06:42:08.350 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-27T06:42:08.406 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-joblib. 2024-05-27T06:42:08.419 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-27T06:42:08.464 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T06:42:08.695 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-sklearn. 2024-05-27T06:42:08.709 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-27T06:42:08.761 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T06:42:08.842 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-27T06:42:08.856 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-27T06:42:08.978 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-27T06:42:09.281 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-sklearn. 2024-05-27T06:42:09.295 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-27T06:42:09.331 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T06:42:09.412 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-27T06:42:09.426 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:09.470 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:09.990 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-27T06:42:10.006 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:10.048 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:10.408 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-cachetools. 2024-05-27T06:42:10.423 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-05-27T06:42:10.466 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-27T06:42:10.752 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-27T06:42:10.767 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-27T06:42:10.835 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-27T06:42:10.968 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-cachetools. 2024-05-27T06:42:10.983 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-05-27T06:42:11.018 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-27T06:42:11.146 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-27T06:42:11.162 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-27T06:42:11.196 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T06:42:11.304 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-27T06:42:11.319 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-27T06:42:11.362 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-27T06:42:11.508 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-rsa. 2024-05-27T06:42:11.523 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-05-27T06:42:11.566 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-27T06:42:11.716 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-27T06:42:11.732 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-27T06:42:11.774 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T06:42:11.903 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-google-auth. 2024-05-27T06:42:11.918 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-05-27T06:42:11.961 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-27T06:42:12.282 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-27T06:42:12.297 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-27T06:42:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-rsa. 2024-05-27T06:42:12.331 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T06:42:12.335 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-05-27T06:42:12.386 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-27T06:42:12.610 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-websocket. 2024-05-27T06:42:12.625 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-05-27T06:42:12.656 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-google-auth. 2024-05-27T06:42:12.672 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-05-27T06:42:12.676 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-27T06:42:12.706 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-27T06:42:12.979 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-27T06:42:12.994 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-27T06:42:13.009 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-27T06:42:13.025 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-27T06:42:13.068 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T06:42:13.128 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T06:42:13.404 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-websocket. 2024-05-27T06:42:13.420 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-05-27T06:42:13.462 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-27T06:42:13.766 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-27T06:42:13.781 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-27T06:42:13.816 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T06:42:14.026 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-27T06:42:14.041 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:14.085 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:14.404 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-27T06:42:14.419 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-27T06:42:14.463 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T06:42:14.496 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-27T06:42:14.510 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:14.554 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:14.816 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-27T06:42:14.831 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T06:42:14.865 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T06:42:14.882 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-27T06:42:14.896 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-27T06:42:14.940 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T06:42:15.085 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package jq. 2024-05-27T06:42:15.100 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T06:42:15.143 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-27T06:42:15.277 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-27T06:42:15.291 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T06:42:15.327 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T06:42:15.396 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package socat. 2024-05-27T06:42:15.411 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-27T06:42:15.454 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-27T06:42:15.571 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package jq. 2024-05-27T06:42:15.585 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-27T06:42:15.629 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-27T06:42:15.774 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package xmlstarlet. 2024-05-27T06:42:15.790 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-27T06:42:15.832 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-27T06:42:15.866 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package socat. 2024-05-27T06:42:15.880 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-27T06:42:15.916 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-27T06:42:16.178 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-test. 2024-05-27T06:42:16.192 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:16.202 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package xmlstarlet. 2024-05-27T06:42:16.216 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-27T06:42:16.236 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-test (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:16.260 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-27T06:42:16.624 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-test. 2024-05-27T06:42:16.638 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:16.673 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-test (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:18.407 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ceph-volume. 2024-05-27T06:42:18.422 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:18.649 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ceph-volume (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:18.769 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ceph-volume. 2024-05-27T06:42:18.785 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ceph-volume (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:19.020 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cephfs-mirror. 2024-05-27T06:42:19.035 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:19.077 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cephfs-mirror (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:19.242 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephfs-mirror. 2024-05-27T06:42:19.257 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:19.301 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephfs-mirror (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:19.398 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-colorama. 2024-05-27T06:42:19.413 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-05-27T06:42:19.447 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-05-27T06:42:19.603 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-colorama. 2024-05-27T06:42:19.618 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-05-27T06:42:19.661 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-05-27T06:42:19.692 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cephfs-shell. 2024-05-27T06:42:19.707 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:19.750 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cephfs-shell (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:19.948 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephfs-shell. 2024-05-27T06:42:19.962 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:19.986 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package cephfs-top. 2024-05-27T06:42:20.001 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:20.005 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephfs-shell (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:20.044 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking cephfs-top (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:20.283 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package cephfs-top. 2024-05-27T06:42:20.297 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3872-g38f9d900-1jammy_all.deb ... 2024-05-27T06:42:20.321 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-27T06:42:20.337 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:20.350 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking cephfs-top (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:20.379 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libcephfs-dev (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:20.603 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-27T06:42:20.617 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:20.661 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libcephfs-dev (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:20.683 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-27T06:42:20.698 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-27T06:42:20.732 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T06:42:20.980 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-27T06:42:20.996 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-27T06:42:21.038 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T06:42:21.062 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-27T06:42:21.072 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-05-27T06:42:21.112 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-27T06:42:21.385 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-27T06:42:21.394 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-05-27T06:42:21.434 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-27T06:42:21.466 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package nvme-cli. 2024-05-27T06:42:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-27T06:42:21.515 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T06:42:21.814 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package nvme-cli. 2024-05-27T06:42:21.826 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-27T06:42:21.864 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T06:42:21.946 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package pkg-config. 2024-05-27T06:42:21.955 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-27T06:42:21.996 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T06:42:22.259 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-27T06:42:22.268 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-27T06:42:22.269 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package pkg-config. 2024-05-27T06:42:22.279 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-27T06:42:22.309 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-27T06:42:22.318 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T06:42:22.547 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-27T06:42:22.557 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-27T06:42:22.613 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-27T06:42:22.712 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-27T06:42:22.720 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-27T06:42:22.762 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-27T06:42:22.975 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-27T06:42:22.987 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-27T06:42:23.016 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-27T06:42:23.048 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pastescript. 2024-05-27T06:42:23.056 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-05-27T06:42:23.089 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-27T06:42:23.303 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pastescript. 2024-05-27T06:42:23.316 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-05-27T06:42:23.361 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-27T06:42:23.393 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pluggy. 2024-05-27T06:42:23.401 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-27T06:42:23.443 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-27T06:42:23.779 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-psutil. 2024-05-27T06:42:23.787 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-27T06:42:23.790 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pluggy. 2024-05-27T06:42:23.801 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-27T06:42:23.828 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-27T06:42:23.839 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-27T06:42:24.148 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-py. 2024-05-27T06:42:24.156 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-05-27T06:42:24.159 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-psutil. 2024-05-27T06:42:24.173 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-27T06:42:24.198 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-27T06:42:24.218 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-27T06:42:24.518 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pygments. 2024-05-27T06:42:24.526 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-27T06:42:24.568 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T06:42:24.579 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-py. 2024-05-27T06:42:24.591 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-05-27T06:42:24.628 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-27T06:42:24.965 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pygments. 2024-05-27T06:42:24.978 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-27T06:42:25.005 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-27T06:42:25.014 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-27T06:42:25.017 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T06:42:25.071 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-27T06:42:25.332 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-toml. 2024-05-27T06:42:25.340 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-05-27T06:42:25.382 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-27T06:42:25.430 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-27T06:42:25.440 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-27T06:42:25.469 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-27T06:42:25.685 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-pytest. 2024-05-27T06:42:25.693 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-27T06:42:25.735 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T06:42:25.772 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-toml. 2024-05-27T06:42:25.784 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-05-27T06:42:25.822 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-27T06:42:26.072 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package python3-simplejson. 2024-05-27T06:42:26.080 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-27T06:42:26.121 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-27T06:42:26.124 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-pytest. 2024-05-27T06:42:26.137 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-27T06:42:26.175 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T06:42:26.382 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-27T06:42:26.393 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-27T06:42:26.433 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-27T06:42:26.570 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package python3-simplejson. 2024-05-27T06:42:26.582 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-27T06:42:26.620 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-27T06:42:26.873 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-27T06:42:26.885 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-27T06:42:26.922 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-27T06:42:27.103 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package radosgw. 2024-05-27T06:42:27.112 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../122-radosgw_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:27.153 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking radosgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:27.669 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package radosgw. 2024-05-27T06:42:27.681 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../122-radosgw_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:27.718 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking radosgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:27.998 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package rbd-fuse. 2024-05-27T06:42:28.008 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:28.047 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking rbd-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:28.338 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-27T06:42:28.595 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T06:42:28.671 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package rbd-fuse. 2024-05-27T06:42:28.685 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3872-g38f9d900-1jammy_amd64.deb ... 2024-05-27T06:42:28.705 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T06:42:28.729 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking rbd-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:28.958 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-27T06:42:29.030 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-27T06:42:29.302 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-27T06:42:29.363 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-27T06:42:29.403 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-27T06:42:29.647 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-27T06:42:30.079 INFO:teuthology.orchestra.run.smithi106.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-27T06:42:30.094 INFO:teuthology.orchestra.run.smithi106.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-05-27T06:42:30.106 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-27T06:42:30.139 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:30.708 INFO:teuthology.orchestra.run.smithi106.stdout:Adding system user cephadm....done 2024-05-27T06:42:30.732 INFO:teuthology.orchestra.run.smithi081.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-27T06:42:30.769 INFO:teuthology.orchestra.run.smithi081.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-05-27T06:42:30.812 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:30.828 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T06:42:31.105 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-27T06:42:31.340 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user cephadm....done 2024-05-27T06:42:31.381 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-27T06:42:31.494 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-27T06:42:31.507 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-27T06:42:31.769 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-27T06:42:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-27T06:42:32.060 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T06:42:32.063 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-27T06:42:32.185 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-27T06:42:32.189 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-27T06:42:32.449 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-27T06:42:32.496 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T06:42:32.691 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-27T06:42:32.851 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-27T06:42:32.864 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-27T06:42:33.157 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-05-27T06:42:33.161 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-27T06:42:33.425 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-27T06:42:33.529 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-27T06:42:33.709 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-27T06:42:33.781 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-05-27T06:42:33.986 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-27T06:42:34.023 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-27T06:42:34.292 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-27T06:42:34.395 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-27T06:42:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-27T06:42:34.646 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-27T06:42:34.889 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-ceph-argparse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:34.928 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-27T06:42:35.165 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T06:42:35.262 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-27T06:42:35.291 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T06:42:35.425 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-05-27T06:42:35.530 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-argparse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:35.701 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T06:42:35.789 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-27T06:42:35.819 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-27T06:42:35.890 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-27T06:42:35.992 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-05-27T06:42:36.078 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-27T06:42:36.251 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-27T06:42:36.411 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-27T06:42:36.467 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-27T06:42:36.500 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-27T06:42:36.598 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T06:42:36.687 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-27T06:42:36.724 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-27T06:42:37.092 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:37.219 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-27T06:42:37.226 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-27T06:42:37.250 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-27T06:42:37.340 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-27T06:42:37.441 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-27T06:42:37.502 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T06:42:37.793 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:37.795 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-27T06:42:37.919 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-27T06:42:38.096 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up zip (3.0-12build2) ... 2024-05-27T06:42:38.195 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-27T06:42:38.222 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T06:42:38.488 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-27T06:42:38.814 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up zip (3.0-12build2) ... 2024-05-27T06:42:38.908 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T06:42:38.940 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-27T06:42:39.175 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T06:42:39.335 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-27T06:42:39.461 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T06:42:39.609 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-27T06:42:39.770 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-27T06:42:39.877 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-27T06:42:40.002 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-27T06:42:40.154 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-27T06:42:40.179 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-27T06:42:40.439 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T06:42:40.472 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-27T06:42:40.832 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T06:42:40.881 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-27T06:42:41.132 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-27T06:42:41.133 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-05-27T06:42:41.468 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T06:42:41.509 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-27T06:42:41.811 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-05-27T06:42:41.828 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-27T06:42:42.079 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-27T06:42:42.153 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-27T06:42:42.196 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:42.498 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T06:42:42.513 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-27T06:42:42.849 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T06:42:43.156 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-27T06:42:43.299 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:43.592 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-05-27T06:42:43.943 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-27T06:42:44.105 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T06:42:44.331 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:44.465 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-27T06:42:44.741 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T06:42:44.867 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-27T06:42:45.002 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T06:42:45.174 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-27T06:42:45.303 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-27T06:42:45.426 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:45.560 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-27T06:42:45.580 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-27T06:42:45.836 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-27T06:42:45.847 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:45.962 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-27T06:42:45.973 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-27T06:42:46.088 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-27T06:42:46.283 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-27T06:42:46.373 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-27T06:42:46.551 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-27T06:42:46.657 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-27T06:42:46.827 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-27T06:42:46.917 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:47.043 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-27T06:42:47.086 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-27T06:42:47.329 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-27T06:42:47.353 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-27T06:42:47.605 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T06:42:47.621 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-27T06:42:47.730 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-27T06:42:47.897 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-27T06:42:48.015 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T06:42:48.141 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T06:42:48.173 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-27T06:42:48.442 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-27T06:42:48.457 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-27T06:42:48.725 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-27T06:42:48.752 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-27T06:42:48.859 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-27T06:42:49.019 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T06:42:49.120 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-27T06:42:49.152 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-27T06:42:49.278 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-27T06:42:49.605 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-27T06:42:49.621 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:49.710 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-27T06:42:49.906 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-27T06:42:49.948 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T06:42:50.074 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T06:42:50.165 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-27T06:42:50.266 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-27T06:42:50.417 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-27T06:42:50.476 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-27T06:42:50.506 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-27T06:42:50.744 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-27T06:42:50.744 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-27T06:42:50.853 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-27T06:42:50.870 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-27T06:42:51.112 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:51.238 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T06:42:51.272 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-27T06:42:51.539 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T06:42:51.540 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-27T06:42:51.657 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-27T06:42:51.857 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T06:42:51.908 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-27T06:42:52.034 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-27T06:42:52.158 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-27T06:42:52.353 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-27T06:42:52.377 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-27T06:42:52.629 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-27T06:42:52.687 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-27T06:42:52.905 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T06:42:52.980 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-27T06:42:53.198 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-27T06:42:53.207 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-27T06:42:53.475 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-27T06:42:53.583 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-27T06:42:53.752 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-27T06:42:53.885 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T06:42:54.028 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-27T06:42:54.212 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T06:42:54.338 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librados2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:54.405 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-27T06:42:54.464 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:54.590 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-27T06:42:54.673 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-27T06:42:54.927 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libcephfs2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.017 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-27T06:42:55.027 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libradosstriper1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.152 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librados2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.154 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-27T06:42:55.278 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.404 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-27T06:42:55.451 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librbd1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.565 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-modules-core (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.707 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.741 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.891 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libradosstriper1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T06:42:55.916 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T06:42:56.017 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-27T06:42:56.311 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librbd1 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.454 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-modules-core (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.578 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libcephfs-dev (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.580 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.718 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rados (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.805 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T06:42:56.805 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-27T06:42:56.844 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up librgw2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:56.970 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cephfs-top (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:57.079 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T06:42:57.514 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libcephfs-dev (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:57.674 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rados (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:57.800 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up librgw2 (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:57.926 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephfs-top (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.052 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-27T06:42:58.341 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rbd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.475 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up rbd-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.601 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-rgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.727 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up python3-cephfs (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.853 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cephfs-shell (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:58.979 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.331 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rbd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.482 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up rbd-fuse (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.532 INFO:teuthology.orchestra.run.smithi106.stdout:Adding group ceph....done 2024-05-27T06:42:59.608 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-rgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.734 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up python3-cephfs (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.860 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephfs-shell (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:42:59.908 INFO:teuthology.orchestra.run.smithi106.stdout:Adding system user ceph....done 2024-05-27T06:42:59.986 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-common (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:00.077 INFO:teuthology.orchestra.run.smithi106.stdout:Setting system user ceph properties....done 2024-05-27T06:43:00.096 INFO:teuthology.orchestra.run.smithi106.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-27T06:43:00.236 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-27T06:43:00.464 INFO:teuthology.orchestra.run.smithi081.stdout:Adding group ceph....done 2024-05-27T06:43:00.630 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-27T06:43:00.841 INFO:teuthology.orchestra.run.smithi081.stdout:Adding system user ceph....done 2024-05-27T06:43:01.009 INFO:teuthology.orchestra.run.smithi081.stdout:Setting system user ceph properties....done 2024-05-27T06:43:01.029 INFO:teuthology.orchestra.run.smithi081.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-27T06:43:01.169 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-27T06:43:01.322 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up cephfs-mirror (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:01.529 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:43:01.530 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:43:01.578 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-27T06:43:02.181 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-test (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:02.307 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up radosgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:02.363 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up cephfs-mirror (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:02.560 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:43:02.560 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-05-27T06:43:02.845 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:43:02.846 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:43:03.517 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-base (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:03.664 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-test (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:03.770 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up radosgw (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:03.850 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-27T06:43:04.301 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:43:04.301 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-27T06:43:04.562 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mds (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:04.956 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-base (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:05.080 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T06:43:05.081 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T06:43:05.344 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-27T06:43:05.755 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:05.971 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T06:43:05.971 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T06:43:06.016 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mds (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:06.248 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T06:43:06.248 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-27T06:43:06.601 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-osd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:06.910 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:06.916 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T06:43:06.916 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T06:43:07.123 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T06:43:07.123 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-27T06:43:07.528 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:07.654 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:07.770 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-osd (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:07.821 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mon (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:08.030 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T06:43:08.030 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T06:43:08.108 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T06:43:08.108 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-27T06:43:08.698 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:08.755 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:08.841 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:08.864 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:08.966 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-mgr-dashboard (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:09.031 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mon (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:09.125 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ceph-volume (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:09.241 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T06:43:09.241 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-27T06:43:09.487 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T06:43:09.604 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T06:43:09.748 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T06:43:09.866 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-cephadm (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:10.018 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:10.144 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-mgr-dashboard (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:10.262 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ceph-volume (19.0.0-3872-g38f9d900-1jammy) ... 2024-05-27T06:43:10.622 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T06:43:10.732 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T06:43:10.875 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T06:43:10.959 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T06:43:11.648 INFO:teuthology.orchestra.run.smithi106.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T06:43:11.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-05-27T06:43:11.726 INFO:teuthology.orchestra.run.smithi106.stdout:Reading package lists... 2024-05-27T06:43:11.901 INFO:teuthology.orchestra.run.smithi106.stdout:Building dependency tree... 2024-05-27T06:43:11.902 INFO:teuthology.orchestra.run.smithi106.stdout:Reading state information... 2024-05-27T06:43:12.018 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T06:43:12.077 INFO:teuthology.orchestra.run.smithi106.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T06:43:12.078 INFO:teuthology.orchestra.run.smithi106.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T06:43:12.078 INFO:teuthology.orchestra.run.smithi106.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T06:43:12.078 INFO:teuthology.orchestra.run.smithi106.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout:The following additional packages will be installed: 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-05-27T06:43:12.079 INFO:teuthology.orchestra.run.smithi106.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T06:43:12.081 INFO:teuthology.orchestra.run.smithi106.stdout:Suggested packages: 2024-05-27T06:43:12.081 INFO:teuthology.orchestra.run.smithi106.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-05-27T06:43:12.081 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-05-27T06:43:12.081 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-05-27T06:43:12.137 INFO:teuthology.orchestra.run.smithi106.stdout:The following NEW packages will be installed: 2024-05-27T06:43:12.137 INFO:teuthology.orchestra.run.smithi106.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-05-27T06:43:12.137 INFO:teuthology.orchestra.run.smithi106.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-05-27T06:43:12.138 INFO:teuthology.orchestra.run.smithi106.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-05-27T06:43:12.138 INFO:teuthology.orchestra.run.smithi106.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-05-27T06:43:12.138 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-05-27T06:43:12.138 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-05-27T06:43:12.138 INFO:teuthology.orchestra.run.smithi106.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T06:43:12.139 INFO:teuthology.orchestra.run.smithi106.stdout:The following packages will be upgraded: 2024-05-27T06:43:12.140 INFO:teuthology.orchestra.run.smithi106.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-05-27T06:43:12.202 INFO:teuthology.orchestra.run.smithi106.stdout:4 upgraded, 39 newly installed, 0 to remove and 306 not upgraded. 2024-05-27T06:43:12.203 INFO:teuthology.orchestra.run.smithi106.stdout:Need to get 55.7 MB of archives. 2024-05-27T06:43:12.203 INFO:teuthology.orchestra.run.smithi106.stdout:After this operation, 202 MB of additional disk space will be used. 2024-05-27T06:43:12.203 INFO:teuthology.orchestra.run.smithi106.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-05-27T06:43:12.369 INFO:teuthology.orchestra.run.smithi106.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-05-27T06:43:12.379 INFO:teuthology.orchestra.run.smithi106.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-05-27T06:43:12.409 INFO:teuthology.orchestra.run.smithi106.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-05-27T06:43:12.409 INFO:teuthology.orchestra.run.smithi106.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-05-27T06:43:12.410 INFO:teuthology.orchestra.run.smithi106.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-05-27T06:43:12.418 INFO:teuthology.orchestra.run.smithi106.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-05-27T06:43:12.419 INFO:teuthology.orchestra.run.smithi106.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-05-27T06:43:12.421 INFO:teuthology.orchestra.run.smithi106.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-05-27T06:43:12.422 INFO:teuthology.orchestra.run.smithi106.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-05-27T06:43:12.422 INFO:teuthology.orchestra.run.smithi106.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-05-27T06:43:12.423 INFO:teuthology.orchestra.run.smithi106.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-05-27T06:43:12.433 INFO:teuthology.orchestra.run.smithi106.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-05-27T06:43:12.433 INFO:teuthology.orchestra.run.smithi106.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-05-27T06:43:12.434 INFO:teuthology.orchestra.run.smithi106.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-05-27T06:43:12.454 INFO:teuthology.orchestra.run.smithi106.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-05-27T06:43:12.730 INFO:teuthology.orchestra.run.smithi106.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-05-27T06:43:12.732 INFO:teuthology.orchestra.run.smithi106.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-05-27T06:43:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T06:43:12.733 INFO:teuthology.orchestra.run.smithi106.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-05-27T06:43:12.736 INFO:teuthology.orchestra.run.smithi106.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-05-27T06:43:12.736 INFO:teuthology.orchestra.run.smithi106.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-05-27T06:43:12.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-05-27T06:43:12.740 INFO:teuthology.orchestra.run.smithi106.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-05-27T06:43:12.742 INFO:teuthology.orchestra.run.smithi106.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-05-27T06:43:12.749 INFO:teuthology.orchestra.run.smithi106.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-05-27T06:43:12.750 INFO:teuthology.orchestra.run.smithi106.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-05-27T06:43:12.773 INFO:teuthology.orchestra.run.smithi106.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-05-27T06:43:12.776 INFO:teuthology.orchestra.run.smithi106.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-05-27T06:43:12.787 INFO:teuthology.orchestra.run.smithi106.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-05-27T06:43:12.809 INFO:teuthology.orchestra.run.smithi106.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-05-27T06:43:12.812 INFO:teuthology.orchestra.run.smithi081.stdout:Reading package lists... 2024-05-27T06:43:12.813 INFO:teuthology.orchestra.run.smithi106.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-05-27T06:43:12.814 INFO:teuthology.orchestra.run.smithi106.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-05-27T06:43:12.814 INFO:teuthology.orchestra.run.smithi106.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-05-27T06:43:12.815 INFO:teuthology.orchestra.run.smithi106.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-05-27T06:43:12.831 INFO:teuthology.orchestra.run.smithi106.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-05-27T06:43:12.834 INFO:teuthology.orchestra.run.smithi106.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-05-27T06:43:12.987 INFO:teuthology.orchestra.run.smithi081.stdout:Building dependency tree... 2024-05-27T06:43:12.987 INFO:teuthology.orchestra.run.smithi081.stdout:Reading state information... 2024-05-27T06:43:13.034 INFO:teuthology.orchestra.run.smithi106.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-05-27T06:43:13.034 INFO:teuthology.orchestra.run.smithi106.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-05-27T06:43:13.034 INFO:teuthology.orchestra.run.smithi106.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-05-27T06:43:13.034 INFO:teuthology.orchestra.run.smithi106.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-05-27T06:43:13.035 INFO:teuthology.orchestra.run.smithi106.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-05-27T06:43:13.035 INFO:teuthology.orchestra.run.smithi106.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-05-27T06:43:13.051 INFO:teuthology.orchestra.run.smithi106.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-05-27T06:43:13.051 INFO:teuthology.orchestra.run.smithi106.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-05-27T06:43:13.158 INFO:teuthology.orchestra.run.smithi081.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-05-27T06:43:13.158 INFO:teuthology.orchestra.run.smithi081.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-05-27T06:43:13.158 INFO:teuthology.orchestra.run.smithi081.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-05-27T06:43:13.158 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages were automatically installed and are no longer required: 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-27T06:43:13.159 INFO:teuthology.orchestra.run.smithi081.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-27T06:43:13.160 INFO:teuthology.orchestra.run.smithi081.stdout:The following additional packages will be installed: 2024-05-27T06:43:13.160 INFO:teuthology.orchestra.run.smithi081.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-05-27T06:43:13.160 INFO:teuthology.orchestra.run.smithi081.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-05-27T06:43:13.161 INFO:teuthology.orchestra.run.smithi081.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-05-27T06:43:13.161 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-05-27T06:43:13.161 INFO:teuthology.orchestra.run.smithi081.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T06:43:13.162 INFO:teuthology.orchestra.run.smithi081.stdout:Suggested packages: 2024-05-27T06:43:13.162 INFO:teuthology.orchestra.run.smithi081.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-05-27T06:43:13.162 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-05-27T06:43:13.162 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-05-27T06:43:13.215 INFO:teuthology.orchestra.run.smithi081.stdout:The following NEW packages will be installed: 2024-05-27T06:43:13.216 INFO:teuthology.orchestra.run.smithi081.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-05-27T06:43:13.216 INFO:teuthology.orchestra.run.smithi081.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-05-27T06:43:13.216 INFO:teuthology.orchestra.run.smithi081.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-05-27T06:43:13.217 INFO:teuthology.orchestra.run.smithi081.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-05-27T06:43:13.217 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-05-27T06:43:13.217 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-05-27T06:43:13.217 INFO:teuthology.orchestra.run.smithi081.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-05-27T06:43:13.218 INFO:teuthology.orchestra.run.smithi081.stdout:The following packages will be upgraded: 2024-05-27T06:43:13.218 INFO:teuthology.orchestra.run.smithi081.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-05-27T06:43:13.284 INFO:teuthology.orchestra.run.smithi081.stdout:4 upgraded, 39 newly installed, 0 to remove and 306 not upgraded. 2024-05-27T06:43:13.284 INFO:teuthology.orchestra.run.smithi081.stdout:Need to get 55.7 MB of archives. 2024-05-27T06:43:13.284 INFO:teuthology.orchestra.run.smithi081.stdout:After this operation, 202 MB of additional disk space will be used. 2024-05-27T06:43:13.284 INFO:teuthology.orchestra.run.smithi081.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-05-27T06:43:13.428 INFO:teuthology.orchestra.run.smithi106.stdout:Preconfiguring packages ... 2024-05-27T06:43:13.493 INFO:teuthology.orchestra.run.smithi081.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-05-27T06:43:13.496 INFO:teuthology.orchestra.run.smithi081.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-05-27T06:43:13.536 INFO:teuthology.orchestra.run.smithi081.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-05-27T06:43:13.537 INFO:teuthology.orchestra.run.smithi081.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-05-27T06:43:13.538 INFO:teuthology.orchestra.run.smithi081.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-05-27T06:43:13.549 INFO:teuthology.orchestra.run.smithi081.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-05-27T06:43:13.550 INFO:teuthology.orchestra.run.smithi081.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-05-27T06:43:13.551 INFO:teuthology.orchestra.run.smithi081.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-05-27T06:43:13.552 INFO:teuthology.orchestra.run.smithi081.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-05-27T06:43:13.553 INFO:teuthology.orchestra.run.smithi081.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-05-27T06:43:13.554 INFO:teuthology.orchestra.run.smithi081.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-05-27T06:43:13.560 INFO:teuthology.orchestra.run.smithi081.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-05-27T06:43:13.574 INFO:teuthology.orchestra.run.smithi106.stdout:Fetched 55.7 MB in 1s (61.7 MB/s) 2024-05-27T06:43:13.576 INFO:teuthology.orchestra.run.smithi081.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-05-27T06:43:13.576 INFO:teuthology.orchestra.run.smithi081.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-05-27T06:43:13.577 INFO:teuthology.orchestra.run.smithi081.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-05-27T06:43:13.861 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140920 files and directories currently installed.) 2024-05-27T06:43:13.865 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:13.872 INFO:teuthology.orchestra.run.smithi081.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-05-27T06:43:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-05-27T06:43:13.874 INFO:teuthology.orchestra.run.smithi081.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-05-27T06:43:13.878 INFO:teuthology.orchestra.run.smithi081.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-05-27T06:43:13.879 INFO:teuthology.orchestra.run.smithi081.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-05-27T06:43:13.884 INFO:teuthology.orchestra.run.smithi081.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-05-27T06:43:13.885 INFO:teuthology.orchestra.run.smithi081.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-05-27T06:43:13.894 INFO:teuthology.orchestra.run.smithi081.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-05-27T06:43:13.895 INFO:teuthology.orchestra.run.smithi081.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-05-27T06:43:13.918 INFO:teuthology.orchestra.run.smithi081.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-05-27T06:43:13.919 INFO:teuthology.orchestra.run.smithi081.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-05-27T06:43:13.931 INFO:teuthology.orchestra.run.smithi081.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-05-27T06:43:13.955 INFO:teuthology.orchestra.run.smithi081.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-05-27T06:43:13.960 INFO:teuthology.orchestra.run.smithi081.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-05-27T06:43:13.960 INFO:teuthology.orchestra.run.smithi081.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-05-27T06:43:13.961 INFO:teuthology.orchestra.run.smithi081.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-05-27T06:43:13.962 INFO:teuthology.orchestra.run.smithi081.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-05-27T06:43:13.975 INFO:teuthology.orchestra.run.smithi081.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-05-27T06:43:13.977 INFO:teuthology.orchestra.run.smithi081.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-05-27T06:43:14.023 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:14.172 INFO:teuthology.orchestra.run.smithi081.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-05-27T06:43:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-05-27T06:43:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-05-27T06:43:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-05-27T06:43:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-05-27T06:43:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-05-27T06:43:14.188 INFO:teuthology.orchestra.run.smithi081.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-05-27T06:43:14.188 INFO:teuthology.orchestra.run.smithi081.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-05-27T06:43:14.565 INFO:teuthology.orchestra.run.smithi081.stdout:Preconfiguring packages ... 2024-05-27T06:43:14.640 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:Fetched 55.7 MB in 1s (57.9 MB/s) 2024-05-27T06:43:15.006 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140920 files and directories currently installed.) 2024-05-27T06:43:15.009 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:15.151 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:15.672 INFO:teuthology.orchestra.run.smithi106.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-05-27T06:43:15.767 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:15.945 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package flex. 2024-05-27T06:43:16.014 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140920 files and directories currently installed.) 2024-05-27T06:43:16.018 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:16.060 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking flex (2.6.4-8build2) ... 2024-05-27T06:43:16.579 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T06:43:16.786 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-05-27T06:43:16.854 INFO:teuthology.orchestra.run.smithi081.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-05-27T06:43:17.157 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package flex. 2024-05-27T06:43:17.226 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140920 files and directories currently installed.) 2024-05-27T06:43:17.230 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:17.272 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking flex (2.6.4-8build2) ... 2024-05-27T06:43:17.291 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:17.788 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140993 files and directories currently installed.) 2024-05-27T06:43:17.792 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:17.900 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:17.908 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T06:43:18.128 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-05-27T06:43:18.295 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:18.431 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:18.526 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:18.736 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:19.048 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140993 files and directories currently installed.) 2024-05-27T06:43:19.052 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:19.106 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package bison. 2024-05-27T06:43:19.177 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:19.178 INFO:teuthology.orchestra.run.smithi106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140993 files and directories currently installed.) 2024-05-27T06:43:19.181 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-05-27T06:43:19.225 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T06:43:19.565 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-05-27T06:43:19.643 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-05-27T06:43:19.657 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-05-27T06:43:19.701 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-05-27T06:43:19.701 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T06:43:20.037 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package dump. 2024-05-27T06:43:20.049 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:20.051 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-05-27T06:43:20.096 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking dump (0.4b47-3) ... 2024-05-27T06:43:20.399 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package indent. 2024-05-27T06:43:20.401 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package bison. 2024-05-27T06:43:20.413 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:20.449 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T06:43:20.473 INFO:teuthology.orchestra.run.smithi081.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140993 files and directories currently installed.) 2024-05-27T06:43:20.477 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-05-27T06:43:20.520 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T06:43:20.702 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-05-27T06:43:20.716 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-05-27T06:43:20.760 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T06:43:20.921 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-05-27T06:43:20.935 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-05-27T06:43:20.980 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T06:43:21.089 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-05-27T06:43:21.103 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-05-27T06:43:21.147 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T06:43:21.307 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package dump. 2024-05-27T06:43:21.322 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-05-27T06:43:21.366 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking dump (0.4b47-3) ... 2024-05-27T06:43:21.416 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-05-27T06:43:21.430 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-05-27T06:43:21.475 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T06:43:21.669 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package indent. 2024-05-27T06:43:21.684 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:21.728 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T06:43:21.761 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libfl2:amd64. 2024-05-27T06:43:21.775 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:21.819 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:21.972 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-05-27T06:43:21.987 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-05-27T06:43:22.022 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T06:43:22.072 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-05-27T06:43:22.086 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:22.130 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:22.326 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-05-27T06:43:22.341 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-05-27T06:43:22.385 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T06:43:22.399 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libjson-perl. 2024-05-27T06:43:22.413 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-05-27T06:43:22.458 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-05-27T06:43:22.654 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-05-27T06:43:22.669 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-05-27T06:43:22.704 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T06:43:22.728 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libllvm14:amd64. 2024-05-27T06:43:22.741 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-05-27T06:43:22.777 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T06:43:23.024 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libfl2:amd64. 2024-05-27T06:43:23.039 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:23.082 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:23.318 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-05-27T06:43:23.333 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-05-27T06:43:23.376 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:23.655 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libjson-perl. 2024-05-27T06:43:23.671 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-05-27T06:43:23.713 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-05-27T06:43:24.033 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libllvm14:amd64. 2024-05-27T06:43:24.048 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-05-27T06:43:24.091 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T06:43:24.139 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-05-27T06:43:24.154 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T06:43:24.197 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:24.458 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libmbim-proxy. 2024-05-27T06:43:24.473 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T06:43:24.516 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:24.827 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-05-27T06:43:24.843 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T06:43:24.902 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:25.222 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libndp0:amd64. 2024-05-27T06:43:25.238 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-05-27T06:43:25.280 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T06:43:25.494 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-05-27T06:43:25.510 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T06:43:25.552 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:25.600 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libnm0:amd64. 2024-05-27T06:43:25.615 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T06:43:25.658 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T06:43:25.814 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libmbim-proxy. 2024-05-27T06:43:25.829 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-05-27T06:43:25.872 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:26.036 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libpq5:amd64. 2024-05-27T06:43:26.051 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:26.094 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:26.208 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-05-27T06:43:26.224 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T06:43:26.266 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:26.389 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-05-27T06:43:26.405 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:26.438 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:26.603 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libndp0:amd64. 2024-05-27T06:43:26.618 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-05-27T06:43:26.661 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T06:43:26.750 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libqmi-proxy. 2024-05-27T06:43:26.765 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:26.808 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:26.981 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libnm0:amd64. 2024-05-27T06:43:26.996 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T06:43:27.039 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T06:43:27.052 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-05-27T06:43:27.067 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T06:43:27.110 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:27.392 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libpq5:amd64. 2024-05-27T06:43:27.408 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:27.450 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:27.614 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-05-27T06:43:27.628 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-05-27T06:43:27.664 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T06:43:27.795 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-05-27T06:43:27.810 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:27.853 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:28.008 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package modemmanager. 2024-05-27T06:43:28.023 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T06:43:28.067 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:28.173 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libqmi-proxy. 2024-05-27T06:43:28.188 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:28.231 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:28.433 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-05-27T06:43:28.449 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-05-27T06:43:28.483 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:28.723 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package network-manager. 2024-05-27T06:43:28.739 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T06:43:28.790 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-05-27T06:43:28.987 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-05-27T06:43:29.002 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-05-27T06:43:29.045 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T06:43:29.390 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package modemmanager. 2024-05-27T06:43:29.405 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-05-27T06:43:29.448 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:29.456 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package ppp. 2024-05-27T06:43:29.471 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-05-27T06:43:29.840 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T06:43:30.080 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package network-manager. 2024-05-27T06:43:30.095 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-05-27T06:43:30.135 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package pptp-linux. 2024-05-27T06:43:30.147 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-05-27T06:43:30.149 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-05-27T06:43:30.194 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-05-27T06:43:30.479 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package network-manager-pptp. 2024-05-27T06:43:30.493 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-05-27T06:43:30.538 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-05-27T06:43:30.679 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package ppp. 2024-05-27T06:43:30.694 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-05-27T06:43:30.824 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-client-common. 2024-05-27T06:43:30.838 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-05-27T06:43:30.882 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-client-common (238) ... 2024-05-27T06:43:31.082 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T06:43:31.286 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-client-14. 2024-05-27T06:43:31.300 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:31.344 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:31.368 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package pptp-linux. 2024-05-27T06:43:31.382 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-05-27T06:43:31.418 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-05-27T06:43:31.645 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package network-manager-pptp. 2024-05-27T06:43:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-05-27T06:43:31.712 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-05-27T06:43:31.925 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-common. 2024-05-27T06:43:31.938 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-05-27T06:43:31.999 INFO:teuthology.orchestra.run.smithi106.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-05-27T06:43:32.024 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-client-common. 2024-05-27T06:43:32.030 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-common (238) ... 2024-05-27T06:43:32.038 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-05-27T06:43:32.082 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-client-common (238) ... 2024-05-27T06:43:32.385 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-client-14. 2024-05-27T06:43:32.400 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:32.444 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:32.522 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-14. 2024-05-27T06:43:32.537 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:32.583 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:33.057 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-common. 2024-05-27T06:43:33.072 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-05-27T06:43:33.133 INFO:teuthology.orchestra.run.smithi081.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-05-27T06:43:33.164 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-common (238) ... 2024-05-27T06:43:33.625 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql. 2024-05-27T06:43:33.639 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-05-27T06:43:33.656 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-14. 2024-05-27T06:43:33.671 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-05-27T06:43:33.683 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql (14+238) ... 2024-05-27T06:43:33.717 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:33.960 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-client. 2024-05-27T06:43:33.975 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-05-27T06:43:34.018 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-client (14+238) ... 2024-05-27T06:43:34.245 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package postgresql-contrib. 2024-05-27T06:43:34.260 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-05-27T06:43:34.304 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking postgresql-contrib (14+238) ... 2024-05-27T06:43:34.615 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package tcl8.6. 2024-05-27T06:43:34.629 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-05-27T06:43:34.664 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T06:43:34.934 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package tcl. 2024-05-27T06:43:34.948 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-05-27T06:43:34.996 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-05-27T06:43:35.134 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql. 2024-05-27T06:43:35.149 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-05-27T06:43:35.192 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql (14+238) ... 2024-05-27T06:43:35.345 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package wpasupplicant. 2024-05-27T06:43:35.359 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-05-27T06:43:35.403 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T06:43:35.453 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-client. 2024-05-27T06:43:35.467 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-05-27T06:43:35.511 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-client (14+238) ... 2024-05-27T06:43:35.755 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package postgresql-contrib. 2024-05-27T06:43:35.770 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-05-27T06:43:35.814 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking postgresql-contrib (14+238) ... 2024-05-27T06:43:36.133 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package tcl8.6. 2024-05-27T06:43:36.147 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-05-27T06:43:36.190 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-05-27T06:43:36.191 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T06:43:36.206 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-05-27T06:43:36.248 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-05-27T06:43:36.486 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package tcl. 2024-05-27T06:43:36.500 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-05-27T06:43:36.548 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-05-27T06:43:36.610 INFO:teuthology.orchestra.run.smithi106.stdout:Selecting previously unselected package usb-modeswitch. 2024-05-27T06:43:36.626 INFO:teuthology.orchestra.run.smithi106.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-05-27T06:43:36.668 INFO:teuthology.orchestra.run.smithi106.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T06:43:36.863 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package wpasupplicant. 2024-05-27T06:43:36.878 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-05-27T06:43:36.921 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T06:43:36.937 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-client-common (238) ... 2024-05-27T06:43:37.149 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up flex (2.6.4-8build2) ... 2024-05-27T06:43:37.275 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T06:43:37.301 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-05-27T06:43:37.317 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-05-27T06:43:37.359 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-05-27T06:43:37.401 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:37.527 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T06:43:37.636 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T06:43:37.738 INFO:teuthology.orchestra.run.smithi081.stdout:Selecting previously unselected package usb-modeswitch. 2024-05-27T06:43:37.753 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-05-27T06:43:37.754 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:37.804 INFO:teuthology.orchestra.run.smithi081.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T06:43:38.063 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-client-common (238) ... 2024-05-27T06:43:38.302 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up flex (2.6.4-8build2) ... 2024-05-27T06:43:38.428 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-05-27T06:43:38.554 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:38.663 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-05-27T06:43:38.748 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-05-27T06:43:38.790 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-05-27T06:43:38.851 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T06:43:38.916 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:39.237 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T06:43:39.237 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T06:43:39.875 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-05-27T06:43:39.963 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-05-27T06:43:39.987 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:40.113 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T06:43:40.239 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:40.358 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T06:43:40.359 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-05-27T06:43:40.365 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-05-27T06:43:40.491 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T06:43:40.742 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:40.868 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T06:43:40.994 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T06:43:41.041 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-05-27T06:43:41.167 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-05-27T06:43:41.293 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-05-27T06:43:41.419 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-05-27T06:43:41.545 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-05-27T06:43:41.670 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:41.674 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T06:43:41.780 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-05-27T06:43:41.800 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up dump (0.4b47-3) ... 2024-05-27T06:43:41.895 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-05-27T06:43:41.920 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-05-27T06:43:41.960 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:42.078 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T06:43:42.179 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up tcl (8.6.11+1build2) ... 2024-05-27T06:43:42.296 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libjson-perl (4.04000-1) ... 2024-05-27T06:43:42.456 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T06:43:42.545 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-05-27T06:43:42.595 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-05-27T06:43:42.615 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T06:43:42.721 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up dump (0.4b47-3) ... 2024-05-27T06:43:42.741 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T06:43:42.816 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-05-27T06:43:42.898 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:42.918 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T06:43:43.024 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-05-27T06:43:43.044 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:43.150 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up tcl (8.6.11+1build2) ... 2024-05-27T06:43:43.153 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-05-27T06:43:43.276 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libjson-perl (4.04000-1) ... 2024-05-27T06:43:43.329 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-client (14+238) ... 2024-05-27T06:43:43.402 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-05-27T06:43:43.447 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-05-27T06:43:43.491 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-05-27T06:43:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-05-27T06:43:43.687 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-05-27T06:43:43.864 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-05-27T06:43:43.990 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout:The following network interfaces were found in /etc/network/interfaces 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout:which means they are currently configured by ifupdown: 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout:- enp3s0f1 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout:If you want to manage those interfaces with NetworkManager instead 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout:remove their configuration from /etc/network/interfaces. 2024-05-27T06:43:44.047 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:44.124 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-05-27T06:43:44.180 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-05-27T06:43:44.292 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-client (14+238) ... 2024-05-27T06:43:44.418 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-05-27T06:43:44.601 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-05-27T06:43:45.006 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout:The following network interfaces were found in /etc/network/interfaces 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout:which means they are currently configured by ifupdown: 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout:- enp3s0f1 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout:If you want to manage those interfaces with NetworkManager instead 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout:remove their configuration from /etc/network/interfaces. 2024-05-27T06:43:45.384 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:45.516 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-05-27T06:43:45.691 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:45.851 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:45.950 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-05-27T06:43:45.978 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-05-27T06:43:46.145 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-common (238) ... 2024-05-27T06:43:46.376 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-05-27T06:43:47.104 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-05-27T06:43:47.231 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-05-27T06:43:47.357 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-05-27T06:43:47.483 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-common (238) ... 2024-05-27T06:43:47.657 INFO:teuthology.orchestra.run.smithi106.stdout:Adding user postgres to group ssl-cert 2024-05-27T06:43:48.081 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:48.081 INFO:teuthology.orchestra.run.smithi106.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-05-27T06:43:48.423 INFO:teuthology.orchestra.run.smithi081.stdout:Adding user postgres to group ssl-cert 2024-05-27T06:43:48.471 INFO:teuthology.orchestra.run.smithi106.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-05-27T06:43:48.471 INFO:teuthology.orchestra.run.smithi106.stdout:Removing obsolete dictionary files: 2024-05-27T06:43:48.681 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:48.681 INFO:teuthology.orchestra.run.smithi081.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-05-27T06:43:48.944 INFO:teuthology.orchestra.run.smithi081.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-05-27T06:43:48.944 INFO:teuthology.orchestra.run.smithi081.stdout:Removing obsolete dictionary files: 2024-05-27T06:43:49.186 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-05-27T06:43:49.656 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:49.679 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-05-27T06:43:49.997 INFO:teuthology.orchestra.run.smithi106.stdout:Creating new PostgreSQL cluster 14/main ... 2024-05-27T06:43:50.012 INFO:teuthology.orchestra.run.smithi106.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-05-27T06:43:50.068 INFO:teuthology.orchestra.run.smithi106.stdout:The files belonging to this database system will be owned by user "postgres". 2024-05-27T06:43:50.068 INFO:teuthology.orchestra.run.smithi106.stdout:This user must also own the server process. 2024-05-27T06:43:50.068 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:50.068 INFO:teuthology.orchestra.run.smithi106.stdout:The database cluster will be initialized with locale "en_US". 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:The default text search configuration will be set to "english". 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:Data page checksums are disabled. 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-05-27T06:43:50.069 INFO:teuthology.orchestra.run.smithi106.stdout:creating subdirectories ... ok 2024-05-27T06:43:50.070 INFO:teuthology.orchestra.run.smithi106.stdout:selecting dynamic shared memory implementation ... posix 2024-05-27T06:43:50.093 INFO:teuthology.orchestra.run.smithi106.stdout:selecting default max_connections ... 100 2024-05-27T06:43:50.127 INFO:teuthology.orchestra.run.smithi106.stdout:selecting default shared_buffers ... 128MB 2024-05-27T06:43:50.142 INFO:teuthology.orchestra.run.smithi106.stdout:selecting default time zone ... Etc/UTC 2024-05-27T06:43:50.142 INFO:teuthology.orchestra.run.smithi106.stdout:creating configuration files ... ok 2024-05-27T06:43:50.143 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-05-27T06:43:50.308 INFO:teuthology.orchestra.run.smithi106.stdout:running bootstrap script ... ok 2024-05-27T06:43:50.478 INFO:teuthology.orchestra.run.smithi081.stdout:Creating new PostgreSQL cluster 14/main ... 2024-05-27T06:43:50.499 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-05-27T06:43:50.550 INFO:teuthology.orchestra.run.smithi081.stdout:The files belonging to this database system will be owned by user "postgres". 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:This user must also own the server process. 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:The database cluster will be initialized with locale "en_US". 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:The default text search configuration will be set to "english". 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:Data page checksums are disabled. 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:43:50.551 INFO:teuthology.orchestra.run.smithi081.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-05-27T06:43:50.552 INFO:teuthology.orchestra.run.smithi081.stdout:creating subdirectories ... ok 2024-05-27T06:43:50.552 INFO:teuthology.orchestra.run.smithi081.stdout:selecting dynamic shared memory implementation ... posix 2024-05-27T06:43:50.575 INFO:teuthology.orchestra.run.smithi081.stdout:selecting default max_connections ... 100 2024-05-27T06:43:50.609 INFO:teuthology.orchestra.run.smithi081.stdout:selecting default shared_buffers ... 128MB 2024-05-27T06:43:50.623 INFO:teuthology.orchestra.run.smithi081.stdout:selecting default time zone ... Etc/UTC 2024-05-27T06:43:50.624 INFO:teuthology.orchestra.run.smithi081.stdout:creating configuration files ... ok 2024-05-27T06:43:50.789 INFO:teuthology.orchestra.run.smithi081.stdout:running bootstrap script ... ok 2024-05-27T06:43:50.934 INFO:teuthology.orchestra.run.smithi106.stdout:performing post-bootstrap initialization ... ok 2024-05-27T06:43:51.410 INFO:teuthology.orchestra.run.smithi081.stdout:performing post-bootstrap initialization ... ok 2024-05-27T06:43:51.524 INFO:teuthology.orchestra.run.smithi106.stdout:syncing data to disk ... ok 2024-05-27T06:43:51.974 INFO:teuthology.orchestra.run.smithi081.stdout:syncing data to disk ... ok 2024-05-27T06:43:52.017 INFO:teuthology.orchestra.run.smithi106.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-05-27T06:43:52.491 INFO:teuthology.orchestra.run.smithi081.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-05-27T06:43:55.021 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:55.165 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql-contrib (14+238) ... 2024-05-27T06:43:55.291 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up postgresql (14+238) ... 2024-05-27T06:43:55.416 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:55.534 INFO:teuthology.orchestra.run.smithi106.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:55.542 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:55.668 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql-contrib (14+238) ... 2024-05-27T06:43:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T06:43:55.777 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T06:43:55.944 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up postgresql (14+238) ... 2024-05-27T06:43:56.070 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-05-27T06:43:56.196 INFO:teuthology.orchestra.run.smithi081.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-05-27T06:43:56.441 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T06:43:56.442 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-05-27T06:43:56.519 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T06:43:56.595 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-05-27T06:43:56.948 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T06:43:57.116 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T06:43:57.215 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-27T06:43:57.308 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-05-27T06:43:57.626 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-27T06:43:57.804 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-27T06:43:58.225 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-05-27T06:43:58.325 INFO:teuthology.orchestra.run.smithi106.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T06:43:58.899 INFO:teuthology.orchestra.run.smithi106.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T06:43:58.907 DEBUG:teuthology.parallel:result is None 2024-05-27T06:43:59.149 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-05-27T06:43:59.242 INFO:teuthology.orchestra.run.smithi081.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-27T06:43:59.802 INFO:teuthology.orchestra.run.smithi081.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-27T06:43:59.807 DEBUG:teuthology.parallel:result is None 2024-05-27T06:43:59.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:44:00.036 DEBUG:teuthology.orchestra.run.smithi081:> dpkg-query -W -f '${Version}' ceph 2024-05-27T06:44:00.058 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:44:00.059 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:44:00.059 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872-g38f9d900-1jammy is installed. 2024-05-27T06:44:00.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:44:00.203 DEBUG:teuthology.orchestra.run.smithi106:> dpkg-query -W -f '${Version}' ceph 2024-05-27T06:44:00.227 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:44:00.227 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3872-g38f9d900-1jammy 2024-05-27T06:44:00.227 INFO:teuthology.task.install:The correct ceph version 19.0.0-3872-g38f9d900-1jammy is installed. 2024-05-27T06:44:00.229 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-27T06:44:00.229 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:00.229 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T06:44:00.248 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:00.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-27T06:44:00.292 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-27T06:44:00.292 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:00.292 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T06:44:00.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T06:44:00.371 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:00.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-05-27T06:44:00.391 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-27T06:44:00.451 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-27T06:44:00.451 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:00.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T06:44:00.470 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T06:44:00.535 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:00.535 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-27T06:44:00.555 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-27T06:44:00.619 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-27T06:44:00.619 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:00.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T06:44:00.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T06:44:00.703 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:00.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-05-27T06:44:00.723 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-27T06:44:00.787 INFO:teuthology.run_tasks:Running task ceph... 2024-05-27T06:44:00.867 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-27T06:44:00.867 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-05-27T06:44:00.871 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /var/log/ceph 2024-05-27T06:44:00.891 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-27T06:44:00.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-27T06:44:00.936 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-27T06:44:00.955 INFO:tasks.ceph:Creating extra log directories... 2024-05-27T06:44:00.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-27T06:44:01.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-27T06:44:01.027 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-27T06:44:01.027 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-27T06:44:01.027 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-27T06:44:01.027 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:01.027 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-27T06:44:01.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-27T06:44:01.153 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:01.154 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-27T06:44:01.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-27T06:44:01.256 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-27T06:44:01.256 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-27T06:44:01.256 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620', 'branch': 'wip-vshankar-testing-20240521.063558-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7727620', 'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240521.063558-debug'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240521.063558-debug', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 2499, 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'sleep_before_teardown': 0, 'subset': '573/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240521.063558-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3zEpWVdrcS2uY7H9F+XcSrx2w2N97QYCo3eZC+Z90jeBEI/s2feOGwSFry5E+z8WQ3h/byfEINYrR8+XMp6Dk=', 'smithi106.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPx5MnPAbZWBaZTH296KC5i8OYkcuVnFmXd7j+LaYUDzFTxt5onthKRHp/o9mC4w0ZSVTTYcuaye/XOBDSRIHMU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': 'b9459a26b0888785d05c47deb35abc9d0354ab93'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': '38f9d900c83c63799fbdbe61acc9a11b0d3554a6', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e7ae7580c417d6ab27fda98fecadca5d9f85d63e', 'timestamp': '2024-05-27_05:52:49', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-05-27T06:44:01.257 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-27T06:44:01.260 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-27T06:44:01.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-05-27T06:44:01.312 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m0777 -- /var/run/ceph 2024-05-27T06:44:01.335 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:01.335 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T06:44:01.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T06:44:01.377 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 895 Links: 1 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-05-27 06:37:40.025396293 +0000 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-05-27 06:37:39.753398722 +0000 2024-05-27T06:44:01.425 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-05-27 06:37:39.753398722 +0000 2024-05-27T06:44:01.426 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-05-27 06:37:39.753398722 +0000 2024-05-27T06:44:01.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T06:44:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-05-27T06:44:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-05-27T06:44:01.482 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000688827 s, 743 kB/s 2024-05-27T06:44:01.484 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T06:44:01.534 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-05-27T06:44:01.589 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T06:44:01.589 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 910 Links: 1 2024-05-27T06:44:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-05-27 06:37:40.605391112 +0000 2024-05-27T06:44:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-05-27 06:37:40.337393505 +0000 2024-05-27T06:44:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-05-27 06:37:40.337393505 +0000 2024-05-27T06:44:01.590 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-05-27 06:37:40.337393505 +0000 2024-05-27T06:44:01.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T06:44:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-05-27T06:44:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-05-27T06:44:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000322474 s, 1.6 MB/s 2024-05-27T06:44:01.648 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T06:44:01.699 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 926 Links: 1 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-05-27 06:37:41.193385860 +0000 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-05-27 06:37:40.921388290 +0000 2024-05-27T06:44:01.753 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-05-27 06:37:40.921388290 +0000 2024-05-27T06:44:01.754 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-05-27 06:37:40.921388290 +0000 2024-05-27T06:44:01.754 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T06:44:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-05-27T06:44:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-05-27T06:44:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000316767 s, 1.6 MB/s 2024-05-27T06:44:01.812 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T06:44:01.863 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-05-27T06:44:01.917 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T06:44:01.917 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:01.917 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 946 Links: 1 2024-05-27T06:44:01.917 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:01.917 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-05-27 06:37:41.721381144 +0000 2024-05-27T06:44:01.918 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-05-27 06:37:41.449383573 +0000 2024-05-27T06:44:01.918 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-05-27 06:37:41.449383573 +0000 2024-05-27T06:44:01.918 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-05-27 06:37:41.449383573 +0000 2024-05-27T06:44:01.918 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T06:44:01.974 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-05-27T06:44:01.974 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-05-27T06:44:01.974 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000372997 s, 1.4 MB/s 2024-05-27T06:44:01.976 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T06:44:02.027 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:02.027 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:02.027 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-05-27T06:44:02.035 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-27T06:44:02.036 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-05-27T06:44:02.088 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-27T06:44:02.088 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:02.088 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 903 Links: 1 2024-05-27T06:44:02.088 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:02.089 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-27 06:37:39.816707260 +0000 2024-05-27T06:44:02.089 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-27 06:37:39.548709651 +0000 2024-05-27T06:44:02.089 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-27 06:37:39.548709651 +0000 2024-05-27T06:44:02.089 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-05-27 06:37:39.548709651 +0000 2024-05-27T06:44:02.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-27T06:44:02.153 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-05-27T06:44:02.153 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-05-27T06:44:02.154 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.00041057 s, 1.2 MB/s 2024-05-27T06:44:02.155 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-27T06:44:02.210 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-05-27T06:44:02.259 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 919 Links: 1 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-27 06:37:40.108704655 +0000 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-27 06:37:40.104704691 +0000 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-27 06:37:40.104704691 +0000 2024-05-27T06:44:02.260 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-05-27 06:37:40.104704691 +0000 2024-05-27T06:44:02.260 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-27T06:44:02.322 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-05-27T06:44:02.322 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-05-27T06:44:02.322 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000378707 s, 1.4 MB/s 2024-05-27T06:44:02.324 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-27T06:44:02.378 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 935 Links: 1 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-27 06:37:40.904697554 +0000 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-27 06:37:40.640699910 +0000 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-27 06:37:40.640699910 +0000 2024-05-27T06:44:02.432 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-05-27 06:37:40.640699910 +0000 2024-05-27T06:44:02.433 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-27T06:44:02.494 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-05-27T06:44:02.494 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-05-27T06:44:02.494 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000397218 s, 1.3 MB/s 2024-05-27T06:44:02.496 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-27T06:44:02.549 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 952 Links: 1 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-05-27 06:37:41.468692523 +0000 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-05-27 06:37:41.200694913 +0000 2024-05-27T06:44:02.600 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-05-27 06:37:41.200694913 +0000 2024-05-27T06:44:02.601 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-05-27 06:37:41.200694913 +0000 2024-05-27T06:44:02.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-27T06:44:02.661 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-05-27T06:44:02.662 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-05-27T06:44:02.662 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000397785 s, 1.3 MB/s 2024-05-27T06:44:02.663 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-27T06:44:02.721 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:02.721 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-27T06:44:02.722 INFO:tasks.ceph:Generating config... 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] debug client = 20 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] debug ms = 1 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-27T06:44:02.723 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-27T06:44:02.723 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-27T06:44:02.723 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-27T06:44:02.723 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-27T06:44:02.723 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-05-27T06:44:02.724 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-27T06:44:02.725 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-27T06:44:02.725 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-27T06:44:02.725 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-05-27T06:44:02.725 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-27T06:44:02.725 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-27T06:44:02.726 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-27T06:44:02.727 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-27T06:44:02.727 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-27T06:44:02.727 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-27T06:44:02.727 INFO:tasks.ceph:Setting up mon.a... 2024-05-27T06:44:02.727 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-27T06:44:02.780 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.keyring 2024-05-27T06:44:02.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-27T06:44:02.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T06:44:02.870 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.81'), ('mon.b', '172.21.15.106'), ('mon.c', '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]')] 2024-05-27T06:44:02.870 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.81,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-27T06:44:02.871 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:02.871 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-27T06:44:02.925 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.81 --add b 172.21.15.106 --addv c '[v2:172.21.15.106:3301,v1:172.21.15.106:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:03.021 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:03.021 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: generated fsid cb957b26-d515-43f2-962a-6a97b6477eea 2024-05-27T06:44:03.021 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-05-27T06:44:03.021 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 0 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:fsid cb957b26-d515-43f2-962a-6a97b6477eea 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:last_changed 2024-05-27T06:44:03.031321+0000 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-05-27T06:44:03.031321+0000 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:min_mon_release 17 (quincy) 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:election_strategy: 1 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:1: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:2: [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] mon.c 2024-05-27T06:44:03.022 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-27T06:44:03.025 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-27T06:44:03.033 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cb957b26-d515-43f2-962a-6a97b6477eea... 2024-05-27T06:44:03.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-27T06:44:03.076 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-05-27T06:44:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-05-27T06:44:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.117 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "172.21.15.81,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = cb957b26-d515-43f2-962a-6a97b6477eea 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.118 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.119 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-05-27T06:44:03.120 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression mode = aggressive 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 20 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 20 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay max = 1 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay probability = 0.005 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay type = client mds 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 2500 2024-05-27T06:44:03.121 INFO:teuthology.orchestra.run.smithi081.stdout: osd blocked scrub grace period = 3600 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: osd max markdown count = 1000 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: osd op complaint time = 180 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-05-27T06:44:03.122 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: debug client = 20 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.123 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon down mkfs grace = 300 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: mon op complaint time = 120 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:03.124 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: client mount timeout = 600 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: client_shutdown_timeout = 120 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: debug client = 20 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: rados mon op timeout = 900 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout: rados osd op timeout = 900 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-05-27T06:44:03.125 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.e] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.f] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout:[mds] 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout: debug mds = 20 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout: debug mds balancer = 20 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout: mds bal fragment size max = 10000 2024-05-27T06:44:03.126 INFO:teuthology.orchestra.run.smithi081.stdout: mds bal merge size = 5 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds bal split bits = 3 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds bal split size = 100 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds debug frag = True 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds debug scatterstat = True 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds oft prefetch dirfrags = True 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds op complaint time = 180 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds verify scatter = True 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds_dir_prefetch = True 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mds_sleep_rank_change = 5000000.0 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay max = 1 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay probability = 0.005 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject delay type = client mds osd 2024-05-27T06:44:03.127 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 2500 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi081.stdout: osd op complaint time = 180 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi081.stdout: rados mon op timeout = 900 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi081.stdout: rados osd op timeout = 900 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi106.stdout:[global] 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi106.stdout: chdir = "" 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi106.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi106.stdout: auth supported = cephx 2024-05-27T06:44:03.128 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: filestore xattr use omap = true 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: mon clock drift allowed = 1.000 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: osd crush chooseleaf type = 0 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: auth debug = true 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on old message = true 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: ms die on bug = true 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: mon pg warn max object skew = 0 2024-05-27T06:44:03.129 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: # disable pg_autoscaler by default for new pools 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default size = 2 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow primary affinity = true 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd allow pg remap = true 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on legacy crush tunables = false 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on crush straw calc version zero = false 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on no sortbitwise = false 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on osd down out interval zero = false 2024-05-27T06:44:03.130 INFO:teuthology.orchestra.run.smithi106.stdout: mon warn on too few osds = false 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon_allow_pool_size_one = true 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: osd default data pool replay window = 5 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon allow pool delete = true 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon cluster log file level = debug 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: debug asserts on shutdown = true 2024-05-27T06:44:03.131 INFO:teuthology.orchestra.run.smithi106.stdout: mon health detail to clog = false 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: mon host = "172.21.15.81,172.21.15.106,[v2:172.21.15.106:3301,v1:172.21.15.106:6790]" 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: fsid = cb957b26-d515-43f2-962a-6a97b6477eea 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout:[osd] 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd journal size = 100 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub load threshold = 5.0 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd scrub max interval = 600 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd mclock profile = high_recovery_ops 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd recover clone overlap = true 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd recovery max chunk = 1048576 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.132 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug shutdown = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug op order = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug verify stray on activate = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug trim objects = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd open classes on start = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug pg log writeout = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd deep scrub update digest min age = 30 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: osd map max advance = 10 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: journal zero on create = true 2024-05-27T06:44:03.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: filestore ondisk finisher threads = 3 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: filestore apply finisher threads = 3 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: bdev debug aio = true 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: osd debug misdirected ops = true 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore block size = 96636764160 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore compression mode = aggressive 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: bluestore fsck on mount = True 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluefs = 20 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: debug bluestore = 20 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-05-27T06:44:03.134 INFO:teuthology.orchestra.run.smithi106.stdout: debug osd = 20 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: debug rocksdb = 10 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: mon client directed command retry = 5 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd backfillfull_ratio = 0.85 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd full ratio = 0.9 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd nearfull ratio = 0.8 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay max = 1 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay probability = 0.005 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay type = client mds 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject socket failures = 2500 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: osd blocked scrub grace period = 3600 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: osd failsafe full ratio = 0.95 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: osd max markdown count = 1000 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: osd objectstore = bluestore 2024-05-27T06:44:03.135 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr] 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug mgr = 20 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: mgr/telemetry/nag = false 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout:[mon] 2024-05-27T06:44:03.136 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: debug mon = 20 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: debug paxos = 20 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: debug auth = 20 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon data avail warn = 5 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon mgr mkfs grace = 240 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min pgs per osd = 4 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd reporter subtree level = osd 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon osd prime pg temp = true 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: mon reweight min bytes per osd = 10 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-27T06:44:03.137 INFO:teuthology.orchestra.run.smithi106.stdout: auth mon ticket ttl = 660 # 11m 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: auth service ticket ttl = 240 # 4m 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: # don't complain about insecure global_id in the test suite 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: # 1m isn't quite enough 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon_down_mkfs_grace = 2m 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon_warn_on_filestore_osds = false 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon down mkfs grace = 300 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: mon op complaint time = 120 2024-05-27T06:44:03.138 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout:[client] 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: rgw cache enabled = true 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable ops log = true 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: rgw enable usage log = true 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: client mount timeout = 600 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: client_shutdown_timeout = 120 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: debug client = 20 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.a] 2024-05-27T06:44:03.139 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.b] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mon.c] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.e] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.f] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout:[mds] 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds = 20 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: debug mds balancer = 20 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: debug ms = 1 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: mds bal fragment size max = 10000 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: mds bal merge size = 5 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: mds bal split bits = 3 2024-05-27T06:44:03.140 INFO:teuthology.orchestra.run.smithi106.stdout: mds bal split size = 100 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug frag = True 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds debug scatterstat = True 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds oft prefetch dirfrags = True 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds op complaint time = 180 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds verify scatter = True 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds_dir_prefetch = True 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mds_sleep_rank_change = 5000000.0 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: mon client directed command retry = 5 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay max = 1 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay probability = 0.005 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject delay type = client mds osd 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: ms inject socket failures = 2500 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: osd op complaint time = 180 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: rados mon op timeout = 900 2024-05-27T06:44:03.141 INFO:teuthology.orchestra.run.smithi106.stdout: rados osd op timeout = 900 2024-05-27T06:44:03.143 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-27T06:44:03.143 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-27T06:44:03.250 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-27T06:44:03.251 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:03.251 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-27T06:44:03.259 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:03.259 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-27T06:44:03.312 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:44:03.312 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:03.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-27T06:44:03.313 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T06:44:03.384 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:03.384 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:03.437 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:44:03.437 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:03.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-27T06:44:03.437 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-27T06:44:03.471 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:03.471 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:03.523 INFO:tasks.ceph:Setting up mon nodes... 2024-05-27T06:44:03.524 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-27T06:44:03.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-27T06:44:03.588 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-27T06:44:03.594 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-27T06:44:03.663 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-27T06:44:03.692 INFO:tasks.ceph:Setting up mds nodes... 2024-05-27T06:44:03.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-27T06:44:03.756 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-27T06:44:03.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-27T06:44:03.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-27T06:44:03.885 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-27T06:44:03.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-27T06:44:03.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-05-27T06:44:04.012 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-05-27T06:44:04.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-05-27T06:44:04.037 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-27T06:44:04.107 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-27T06:44:04.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-27T06:44:04.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-27T06:44:04.243 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-27T06:44:04.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-27T06:44:04.267 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-05-27T06:44:04.370 INFO:teuthology.orchestra.run.smithi106.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-05-27T06:44:04.375 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-05-27T06:44:04.394 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-27T06:44:04.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-27T06:44:04.449 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-27T06:44:04.466 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-27T06:44:04.466 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi106.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-27T06:44:04.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-27T06:44:04.485 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:04.485 INFO:tasks.ceph:role: osd.0 2024-05-27T06:44:04.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:44:04.486 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-27T06:44:04.555 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:04.555 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:04.556 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:04.971 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-05-27T06:44:04.974 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-05-27T06:44:04.974 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-27T06:44:05.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-27T06:44:05.092 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:05.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-27T06:44:05.152 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:05.152 INFO:tasks.ceph:role: osd.1 2024-05-27T06:44:05.152 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:44:05.152 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-27T06:44:05.223 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:05.223 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:05.224 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:05.600 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-05-27T06:44:05.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-05-27T06:44:05.603 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-27T06:44:05.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-27T06:44:05.741 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:05.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-27T06:44:05.804 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:05.804 INFO:tasks.ceph:role: osd.2 2024-05-27T06:44:05.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:44:05.804 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:05.872 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:05.873 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:05.873 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:05.873 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:06.271 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-05-27T06:44:06.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-05-27T06:44:06.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-27T06:44:06.359 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-27T06:44:06.417 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:06.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-27T06:44:06.481 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:06.481 INFO:tasks.ceph:role: osd.3 2024-05-27T06:44:06.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T06:44:06.481 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-27T06:44:06.549 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:06.549 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:06.549 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:06.549 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:06.549 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:06.550 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:06.550 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:06.550 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:06.550 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:06.550 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:06.937 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-05-27T06:44:06.940 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-05-27T06:44:06.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-27T06:44:07.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-27T06:44:07.086 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:07.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:07.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.240+0000 7f3bfb3e17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-27T06:44:07.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.244+0000 7f3bfb3e17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-27T06:44:07.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.244+0000 7f3bfb3e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T06:44:07.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.244+0000 7f3bfb3e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T06:44:07.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.244+0000 7f3bfb3e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-27T06:44:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:07.252+0000 7f3bfb3e17c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-27T06:44:10.290 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-27T06:44:10.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-27T06:44:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-27T06:44:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T06:44:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T06:44:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-27T06:44:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:10.428+0000 7f37e64957c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-27T06:44:13.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-27T06:44:13.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:13.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.608+0000 7f59cc65b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-27T06:44:13.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.608+0000 7f59cc65b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-27T06:44:13.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.608+0000 7f59cc65b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T06:44:13.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.612+0000 7f59cc65b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T06:44:13.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.612+0000 7f59cc65b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-27T06:44:13.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:13.612+0000 7f59cc65b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-27T06:44:16.626 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-27T06:44:16.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.760+0000 7ff254b2d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-27T06:44:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.760+0000 7ff254b2d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-27T06:44:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.760+0000 7ff254b2d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T06:44:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.760+0000 7ff254b2d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T06:44:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.760+0000 7ff254b2d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-27T06:44:16.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:16.764+0000 7ff254b2d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-27T06:44:19.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-27T06:44:19.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-27T06:44:19.843 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:19.843 INFO:tasks.ceph:role: osd.4 2024-05-27T06:44:19.843 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:44:19.844 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:19.919 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:20.336 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-05-27T06:44:20.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-05-27T06:44:20.340 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-27T06:44:20.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-27T06:44:20.470 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:20.471 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-27T06:44:20.531 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:20.531 INFO:tasks.ceph:role: osd.5 2024-05-27T06:44:20.531 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:44:20.531 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-27T06:44:20.602 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:20.603 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:20.972 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-05-27T06:44:20.975 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-05-27T06:44:20.975 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-27T06:44:21.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-27T06:44:21.111 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:21.112 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-27T06:44:21.175 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:21.175 INFO:tasks.ceph:role: osd.6 2024-05-27T06:44:21.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:44:21.175 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:21.247 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:21.642 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-05-27T06:44:21.646 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-05-27T06:44:21.646 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-27T06:44:21.735 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-27T06:44:21.795 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:21.797 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-27T06:44:21.859 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-27T06:44:21.859 INFO:tasks.ceph:role: osd.7 2024-05-27T06:44:21.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T06:44:21.859 DEBUG:teuthology.orchestra.run.smithi106:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-27T06:44:21.930 INFO:teuthology.orchestra.run.smithi106.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-27T06:44:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-27T06:44:21.930 INFO:teuthology.orchestra.run.smithi106.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: = sunit=0 swidth=0 blks 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-27T06:44:21.931 INFO:teuthology.orchestra.run.smithi106.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-27T06:44:22.325 INFO:teuthology.orchestra.run.smithi106.stdout:Discarding blocks...Done. 2024-05-27T06:44:22.328 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi106.front.sepia.ceph.com -o noatime 2024-05-27T06:44:22.328 DEBUG:teuthology.orchestra.run.smithi106:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-27T06:44:22.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-27T06:44:22.508 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: /sbin/restorecon: command not found 2024-05-27T06:44:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:44:22.510 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:22.657 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.666+0000 7f0020d6e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-27T06:44:22.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.674+0000 7f0020d6e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-27T06:44:22.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.674+0000 7f0020d6e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T06:44:22.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.674+0000 7f0020d6e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T06:44:22.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.674+0000 7f0020d6e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-27T06:44:22.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:22.678+0000 7f0020d6e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-27T06:44:25.746 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-27T06:44:25.767 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:25.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-27T06:44:25.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-27T06:44:25.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T06:44:25.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T06:44:25.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-27T06:44:25.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:25.894+0000 7fd7fb6227c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-27T06:44:28.925 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-27T06:44:28.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:29.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.062+0000 7f269ab677c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-27T06:44:29.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.062+0000 7f269ab677c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-27T06:44:29.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.062+0000 7f269ab677c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T06:44:29.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.062+0000 7f269ab677c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T06:44:29.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.062+0000 7f269ab677c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-27T06:44:29.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:29.066+0000 7f269ab677c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-27T06:44:31.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:44:31.262 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:44:32.158 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-27T06:44:32.178 DEBUG:teuthology.orchestra.run.smithi106:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:32.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-27T06:44:32.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-27T06:44:32.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T06:44:32.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T06:44:32.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-27T06:44:32.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:32.294+0000 7ff348a8a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-27T06:44:35.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-27T06:44:35.351 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-27T06:44:35.351 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-27T06:44:35.373 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-27T06:44:35.438 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-27T06:44:35.501 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-05-27T06:44:35.565 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-27T06:44:35.633 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.633 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-27T06:44:35.697 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-27T06:44:35.762 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:35.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-27T06:44:35.830 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:35.830 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-27T06:44:35.851 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:35.851 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-27T06:44:35.915 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:35.915 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-27T06:44:35.978 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:35.978 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-05-27T06:44:36.039 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:36.039 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-27T06:44:36.102 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:36.102 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-27T06:44:36.162 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:36.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-27T06:44:36.222 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:36.222 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-27T06:44:36.282 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:36.282 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-27T06:44:36.290 INFO:tasks.ceph:Adding keys to all mons... 2024-05-27T06:44:36.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-27T06:44:36.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-27T06:44:36.351 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-05-27T06:44:36.351 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCzK1RmVvyqIxAAkfFAYVvB8h3NBJPBjOwahQ== 2024-05-27T06:44:36.351 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-05-27T06:44:36.351 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCzK1RmfMu4LRAAtFFSQQJ/uKlQvhtkc0GJkw== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCzK1Rm9NxZNRAAof/rwlCbdUhaNLaW/Xdz6Q== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.e] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC0K1Rm1GpiARAAbNNfIzeQkINHxy00RunxUQ== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC3K1RmAIq7DhAAPappew/ciscu4Hw19wCF/Q== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC6K1RmffvPGRAAnOYBB2I83plCRIsxNlb5xg== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC9K1Rmv5WvJBAAdIwlNwpzSXnUBJQv+vKc0g== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDAK1RmllGxLRAANlzbZ2btqH1vYgH73KwnUA== 2024-05-27T06:44:36.352 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCzK1RmZsA6KBAA2Cz0ep4x/+xja/KYj7pHfw== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC0K1Rmh/UHBxAAJ6FvyLghwI0UsSNeJ2tkcA== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC0K1Rmq/khDxAA+5SqO6Fa1DGXOxEA5Oqvww== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.f] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC0K1RmblW9FhAAohpjPcnS48JXhzLt7waDxg== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDGK1Rm4JHiJxAAlZvFrpDBPnX4Cu4u8jAEZg== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDJK1RmdMtdNRAAqYQR+P7q/ObpcUEmGLPdeA== 2024-05-27T06:44:36.353 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDNK1RmUTnkAxAA+SCTw063yRJENhZpJlXdOA== 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDQK1RmrwybERAA2MLEniwOwvMp9N2/lZ5rZA== 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQC0K1Rm2gZZGxAAXmNqT2zITwukCTVhmBojHg== 2024-05-27T06:44:36.354 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.y] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQCzK1RmVvyqIxAAkfFAYVvB8h3NBJPBjOwahQ== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.a] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQCzK1RmfMu4LRAAtFFSQQJ/uKlQvhtkc0GJkw== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.c] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQCzK1Rm9NxZNRAAof/rwlCbdUhaNLaW/Xdz6Q== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.e] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC0K1Rm1GpiARAAbNNfIzeQkINHxy00RunxUQ== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.0] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC3K1RmAIq7DhAAPappew/ciscu4Hw19wCF/Q== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.1] 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC6K1RmffvPGRAAnOYBB2I83plCRIsxNlb5xg== 2024-05-27T06:44:36.355 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.2] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC9K1Rmv5WvJBAAdIwlNwpzSXnUBJQv+vKc0g== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.3] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDAK1RmllGxLRAANlzbZ2btqH1vYgH73KwnUA== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[mgr.x] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQCzK1RmZsA6KBAA2Cz0ep4x/+xja/KYj7pHfw== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.b] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC0K1Rmh/UHBxAAJ6FvyLghwI0UsSNeJ2tkcA== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.d] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC0K1Rmq/khDxAA+5SqO6Fa1DGXOxEA5Oqvww== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[mds.f] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC0K1RmblW9FhAAohpjPcnS48JXhzLt7waDxg== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.4] 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDGK1Rm4JHiJxAAlZvFrpDBPnX4Cu4u8jAEZg== 2024-05-27T06:44:36.356 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.5] 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDJK1RmdMtdNRAAqYQR+P7q/ObpcUEmGLPdeA== 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.6] 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDNK1RmUTnkAxAA+SCTw063yRJENhZpJlXdOA== 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout:[osd.7] 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQDQK1RmrwybERAA2MLEniwOwvMp9N2/lZ5rZA== 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout:[client.0] 2024-05-27T06:44:36.357 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQC0K1Rm2gZZGxAAXmNqT2zITwukCTVhmBojHg== 2024-05-27T06:44:36.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T06:44:36.407 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T06:44:36.475 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.478 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.546 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.549 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.618 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:36.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.687 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.750 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.753 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.819 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:36.947 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T06:44:36.950 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-27T06:44:37.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.018 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.089 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-27T06:44:37.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.293 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.364 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-27T06:44:37.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-27T06:44:37.496 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-27T06:44:37.562 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-27T06:44:37.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-27T06:44:37.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T06:44:37.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-27T06:44:37.724 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-27T06:44:37.743 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T06:44:37.866 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-27T06:44:37.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-27T06:44:37.945 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-27T06:44:38.067 DEBUG:teuthology.orchestra.run.smithi106:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-27T06:44:38.086 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:38.089 DEBUG:teuthology.orchestra.run.smithi106:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-27T06:44:38.138 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-27T06:44:38.139 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-27T06:44:38.139 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-27T06:44:38.142 INFO:tasks.ceph.mon.a:Started 2024-05-27T06:44:38.142 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-27T06:44:38.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-27T06:44:38.181 INFO:tasks.ceph.mon.b:Started 2024-05-27T06:44:38.181 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-27T06:44:38.182 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-27T06:44:38.184 INFO:tasks.ceph.mon.c:Started 2024-05-27T06:44:38.184 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-27T06:44:38.185 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-27T06:44:38.185 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-27T06:44:38.188 INFO:tasks.ceph.mgr.y:Started 2024-05-27T06:44:38.188 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-27T06:44:38.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-27T06:44:38.193 INFO:tasks.ceph.mgr.x:Started 2024-05-27T06:44:38.193 INFO:tasks.ceph:Setting crush tunables to default 2024-05-27T06:44:38.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd crush tunables default 2024-05-27T06:44:38.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 Processor -- start 2024-05-27T06:44:38.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 -- start start 2024-05-27T06:44:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 0x7f76f0073e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f00c67f0 con 0x7f76f01595d0 2024-05-27T06:44:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76f00726a0 con 0x7f76f00c6bd0 2024-05-27T06:44:38.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76f6600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f76f0072450 con 0x7f76f015a230 2024-05-27T06:44:38.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76ef400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f76f01595d0 0x7f76f00c77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52060/0 (socket says 172.21.15.81:52060) 2024-05-27T06:44:38.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.356+0000 7f76ef400640 1 -- 172.21.15.81:0/2450286034 learned_addr learned my addr 172.21.15.81:0/2450286034 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:44:38.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.360+0000 7f76eea00640 1 --2- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 0x7f76f0073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.360+0000 7f76eea00640 1 -- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 msgr2=0x7f76f0073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-27T06:44:38.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.360+0000 7f76eea00640 1 -- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 msgr2=0x7f76f0073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-27T06:44:38.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.360+0000 7f76eea00640 1 --2- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 0x7f76f0073e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-27T06:44:38.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.360+0000 7f76eea00640 1 --2- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 0x7f76f0073e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T06:44:38.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4151431978 0 0) 0x7f76f00c67f0 con 0x7f76f01595d0 2024-05-27T06:44:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76d4003530 con 0x7f76f01595d0 2024-05-27T06:44:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 740557541 0 0) 0x7f76f00726a0 con 0x7f76f00c6bd0 2024-05-27T06:44:38.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76f00c67f0 con 0x7f76f00c6bd0 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1290590955 0 0) 0x7f76f00c67f0 con 0x7f76f00c6bd0 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 msgr2=0x7f76f0073e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 --2- 172.21.15.81:0/2450286034 >> v2:172.21.15.106:3300/0 conn(0x7f76f015a230 0x7f76f0073e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 >> v1:172.21.15.81:6789/0 conn(0x7f76f01595d0 legacy=0x7f76f00c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f00c5080 con 0x7f76f00c6bd0 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f76e8001a60 con 0x7f76f00c6bd0 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.516+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 >> v1:172.21.15.106:6789/0 conn(0x7f76f00c6bd0 legacy=0x7f76f00743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76d4004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d4004810 0x7f76d4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76d4005320 0x7f76d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76f0072450 con 0x7f76d4004810 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f76d400e4a0 con 0x7f76d4003c40 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76d400e6a0 con 0x7f76d4005320 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76d4004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76eea00640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d4004810 0x7f76d4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76efe00640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76d4005320 0x7f76d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 -- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76d4005320 msgr2=0x7f76d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76d4005320 0x7f76d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 -- 172.21.15.81:0/2450286034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d4004810 msgr2=0x7f76d4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d4004810 0x7f76d4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76ef400640 1 -- 172.21.15.81:0/2450286034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f00c5080 con 0x7f76d4003c40 2024-05-27T06:44:38.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.520+0000 7f76efe00640 1 -- 172.21.15.81:0/2450286034 reap_dead start 2024-05-27T06:44:38.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76ef400640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76d4004090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76e0001a80 tx=0x7f76e001c600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67f6fc6ef09377f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:38.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e00016a0 con 0x7f76d4003c40 2024-05-27T06:44:38.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e00048b0 con 0x7f76d4003c40 2024-05-27T06:44:38.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76ee000640 1 -- 172.21.15.81:0/2450286034 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0004ca0 con 0x7f76d4003c40 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 -- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 msgr2=0x7f76d4004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76d4004090 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76e0001a80 tx=0x7f76e001c600 comp rx=0 tx=0).stop 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 -- 172.21.15.81:0/2450286034 shutdown_connections 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 --2- 172.21.15.81:0/2450286034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76d4004090 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 -- 172.21.15.81:0/2450286034 >> 172.21.15.81:0/2450286034 conn(0x7f76f0070760 msgr2=0x7f76f00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:38.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 -- 172.21.15.81:0/2450286034 shutdown_connections 2024-05-27T06:44:38.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.524+0000 7f76f6600640 1 -- 172.21.15.81:0/2450286034 wait complete. 2024-05-27T06:44:38.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.528+0000 7f76f6600640 1 Processor -- start 2024-05-27T06:44:38.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.528+0000 7f76f6600640 1 -- start start 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76f01e2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76f01f29c0 0x7f76f01442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76f00c7b70 con 0x7f76f0151df0 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f76f00c79f0 con 0x7f76d4003c40 2024-05-27T06:44:38.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76f00c7cf0 con 0x7f76f01f29c0 2024-05-27T06:44:38.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76ef400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76f01e2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44258/0 (socket says 172.21.15.81:44258) 2024-05-27T06:44:38.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 -- 172.21.15.81:0/3510076881 learned_addr learned my addr 172.21.15.81:0/3510076881 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:38.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76efe00640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76f01f29c0 0x7f76f01442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:38.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 -- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76f01f29c0 msgr2=0x7f76f01442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76f01f29c0 0x7f76f01442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 -- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 msgr2=0x7f76f01e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76f01e2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 -- 172.21.15.81:0/3510076881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f0150bb0 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76eea00640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f76dc011590 tx=0x7f76dc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc019070 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76dc015890 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc00f040 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f0150e90 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f0151450 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f76dc01c070 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f76dc013740 con 0x7f76f0151df0 2024-05-27T06:44:38.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.532+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f006a560 con 0x7f76f0151df0 2024-05-27T06:44:38.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.544+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f76dc013940 con 0x7f76f0151df0 2024-05-27T06:44:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.604+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f76f0075360 con 0x7f76f0151df0 2024-05-27T06:44:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76e7400640 1 -- 172.21.15.81:0/3510076881 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f76dc013b20 con 0x7f76f0151df0 2024-05-27T06:44:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-05-27T06:44:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 msgr2=0x7f76f01f2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f76dc011590 tx=0x7f76dc011a50 comp rx=0 tx=0).stop 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 shutdown_connections 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76f01f29c0 0x7f76f01442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76f0151df0 0x7f76f01f2480 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 --2- 172.21.15.81:0/3510076881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d4003c40 0x7f76f01e2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 >> 172.21.15.81:0/3510076881 conn(0x7f76f0070760 msgr2=0x7f76f0074760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 shutdown_connections 2024-05-27T06:44:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:38.608+0000 7f76f6600640 1 -- 172.21.15.81:0/3510076881 wait complete. 2024-05-27T06:44:38.619 INFO:tasks.ceph:check_enable_crimson: False 2024-05-27T06:44:38.619 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-27T06:44:38.619 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:38.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-27T06:44:38.638 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:38.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-27T06:44:38.701 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:38.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-27T06:44:38.723 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:38.732+0000 7fdcf602a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:44:38.729 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:38.738+0000 7fa2a8e28180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-27T06:44:38.742 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:44:38.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-27T06:44:38.805 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:38.805 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-27T06:44:38.825 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:38.825 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-27T06:44:38.886 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:38.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-27T06:44:38.946 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-05-27T06:44:38.946 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-27T06:44:39.006 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 990faba2-c472-4f9d-977d-aad42731b136 0 2024-05-27T06:44:39.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 Processor -- start 2024-05-27T06:44:39.193 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 -- start start 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa0800cbf80 0x7fa08015cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0800d0510 con 0x7fa080156bd0 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08006e130 con 0x7fa08015d100 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa085c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa0800d0250 con 0x7fa0800cbf80 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa07f400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fa08015d100 0x7fa080167250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:33240/0 (socket says 172.21.15.106:33240) 2024-05-27T06:44:39.195 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa07f400640 1 -- 172.21.15.106:0/2230945088 learned_addr learned my addr 172.21.15.106:0/2230945088 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-05-27T06:44:39.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.202+0000 7fa077e00640 1 --2- 172.21.15.106:0/2230945088 >> v2:172.21.15.81:3300/0 conn(0x7fa0800cbf80 0x7fa08015cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169712401 0 0) 0x7fa08006e130 con 0x7fa08015d100 2024-05-27T06:44:39.196 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa064003530 con 0x7fa08015d100 2024-05-27T06:44:39.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa077e00640 1 -- 172.21.15.106:0/2230945088 >> v1:172.21.15.106:6789/0 conn(0x7fa08015d100 legacy=0x7fa080167250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa077e00640 1 -- 172.21.15.106:0/2230945088 >> v1:172.21.15.81:6789/0 conn(0x7fa080156bd0 legacy=0x7fa0800cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551418354 0 0) 0x7fa0800d0510 con 0x7fa080156bd0 2024-05-27T06:44:39.197 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa077e00640 1 -- 172.21.15.106:0/2230945088 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08006e3d0 con 0x7fa0800cbf80 2024-05-27T06:44:39.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa077e00640 1 --2- 172.21.15.106:0/2230945088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0800cbf80 0x7fa08015cbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097c0 tx=0x7fa06801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe5721a21ba0150 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0680a7070 con 0x7fa0800cbf80 2024-05-27T06:44:39.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0680b0070 con 0x7fa0800cbf80 2024-05-27T06:44:39.199 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.206+0000 7fa07e000640 1 -- 172.21.15.106:0/2230945088 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0680ab890 con 0x7fa0800cbf80 2024-05-27T06:44:39.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- 172.21.15.106:0/2230945088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0800cbf80 msgr2=0x7fa08015cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 --2- 172.21.15.106:0/2230945088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0800cbf80 0x7fa08015cbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097c0 tx=0x7fa06801c470 comp rx=0 tx=0).stop 2024-05-27T06:44:39.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- 172.21.15.106:0/2230945088 shutdown_connections 2024-05-27T06:44:39.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 --2- 172.21.15.106:0/2230945088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0800cbf80 0x7fa08015cbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.200 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- 172.21.15.106:0/2230945088 >> 172.21.15.106:0/2230945088 conn(0x7fa080068cb0 msgr2=0x7fa080072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- 172.21.15.106:0/2230945088 shutdown_connections 2024-05-27T06:44:39.201 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- 172.21.15.106:0/2230945088 wait complete. 2024-05-27T06:44:39.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 Processor -- start 2024-05-27T06:44:39.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.210+0000 7fa085c00640 1 -- start start 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0800cbf80 0x7fa080159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa080156bd0 0x7fa080146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa08015d510 con 0x7fa080156bd0 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa0800cff60 con 0x7fa08015d100 2024-05-27T06:44:39.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa080156fe0 con 0x7fa0800cbf80 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07ea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0800cbf80 0x7fa080159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47344/0 (socket says 172.21.15.106:47344) 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 -- 172.21.15.106:0/1564290446 learned_addr learned my addr 172.21.15.106:0/1564290446 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa077e00640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa080156bd0 0x7fa080146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 -- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0800cbf80 msgr2=0x7fa080159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0800cbf80 0x7fa080159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 -- 172.21.15.106:0/1564290446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa080156bd0 msgr2=0x7fa080146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa080156bd0 0x7fa080146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 -- 172.21.15.106:0/1564290446 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa080159960 con 0x7fa08015d100 2024-05-27T06:44:39.206 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa07f400640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa07800caa0 tx=0x7fa07800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07800ac00 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07800ada0 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0801497d0 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078012620 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa080158f00 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa0780127c0 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa078007070 con 0x7fa08015d100 2024-05-27T06:44:39.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.214+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa080148c30 con 0x7fa08015d100 2024-05-27T06:44:39.209 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.218+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa078012c90 con 0x7fa08015d100 2024-05-27T06:44:39.226 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:39.236+0000 7fdcf602a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:44:39.254 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:39.262+0000 7fa2a8e28180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-27T06:44:39.289 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.298+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "990faba2-c472-4f9d-977d-aad42731b136", "id": 0} v 0) -- 0x7fa08006a560 con 0x7fa08015d100 2024-05-27T06:44:39.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa076a00640 1 -- 172.21.15.106:0/1564290446 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "990faba2-c472-4f9d-977d-aad42731b136", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa07801b400 con 0x7fa08015d100 2024-05-27T06:44:39.294 INFO:teuthology.orchestra.run.smithi106.stdout:0 2024-05-27T06:44:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 msgr2=0x7fa080157ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa07800caa0 tx=0x7fa07800cf60 comp rx=0 tx=0).stop 2024-05-27T06:44:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 shutdown_connections 2024-05-27T06:44:39.295 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa08015d100 0x7fa080157ef0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa080156bd0 0x7fa080146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 --2- 172.21.15.106:0/1564290446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa0800cbf80 0x7fa080159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 >> 172.21.15.106:0/1564290446 conn(0x7fa080068cb0 msgr2=0x7fa08014ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 shutdown_connections 2024-05-27T06:44:39.296 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.302+0000 7fa085c00640 1 -- 172.21.15.106:0/1564290446 wait complete. 2024-05-27T06:44:39.314 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 43a3497e-e0ac-484e-afb6-23dcb6fd48d4 1 2024-05-27T06:44:39.447 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:39.456+0000 7fdcf602a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:44:39.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 Processor -- start 2024-05-27T06:44:39.461 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 -- start start 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6bc157030 con 0x7fd6bc074aa0 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6bc160590 con 0x7fd6bc06f0c0 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6c3000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd6bc160870 con 0x7fd6bc15fe30 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6bbe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6bbe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47350/0 (socket says 172.21.15.106:47350) 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.470+0000 7fd6bbe00640 1 -- 172.21.15.106:0/1642707962 learned_addr learned my addr 172.21.15.106:0/1642707962 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bbe00640 1 -- 172.21.15.106:0/1642707962 >> v1:172.21.15.106:6789/0 conn(0x7fd6bc06f0c0 legacy=0x7fd6bc1671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.467 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bbe00640 1 -- 172.21.15.106:0/1642707962 >> v1:172.21.15.81:6789/0 conn(0x7fd6bc074aa0 legacy=0x7fd6bc06e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bbe00640 1 -- 172.21.15.106:0/1642707962 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc160b50 con 0x7fd6bc15fe30 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bb400640 1 -- 172.21.15.106:0/1642707962 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531033438 0 0) 0x7fd6bc157030 con 0x7fd6bc074aa0 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bbe00640 1 --2- 172.21.15.106:0/1642707962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009870 tx=0x7fd6ac01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53164b216ce53802 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bb400640 1 -- 172.21.15.106:0/1642707962 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0b5070 con 0x7fd6bc15fe30 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bb400640 1 -- 172.21.15.106:0/1642707962 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ac0a8a50 con 0x7fd6bc15fe30 2024-05-27T06:44:39.468 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.474+0000 7fd6bb400640 1 -- 172.21.15.106:0/1642707962 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0a8e20 con 0x7fd6bc15fe30 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- 172.21.15.106:0/1642707962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc15fe30 msgr2=0x7fd6bc0ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 --2- 172.21.15.106:0/1642707962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009870 tx=0x7fd6ac01ed50 comp rx=0 tx=0).stop 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- 172.21.15.106:0/1642707962 shutdown_connections 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 --2- 172.21.15.106:0/1642707962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc15fe30 0x7fd6bc0ca4f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- 172.21.15.106:0/1642707962 >> 172.21.15.106:0/1642707962 conn(0x7fd6bc072500 msgr2=0x7fd6bc072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- 172.21.15.106:0/1642707962 shutdown_connections 2024-05-27T06:44:39.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- 172.21.15.106:0/1642707962 wait complete. 2024-05-27T06:44:39.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 Processor -- start 2024-05-27T06:44:39.470 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- start start 2024-05-27T06:44:39.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc06f0c0 0x7fd6bc15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc0caa30 con 0x7fd6bc06f0c0 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc160240 con 0x7fd6bc074aa0 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.478+0000 7fd6c3000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc0cabb0 con 0x7fd6bc15fe30 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47364/0 (socket says 172.21.15.106:47364) 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 -- 172.21.15.106:0/833536671 learned_addr learned my addr 172.21.15.106:0/833536671 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44402/0 (socket says 172.21.15.106:44402) 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6bbe00640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc06f0c0 0x7fd6bc15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 -- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 msgr2=0x7fd6bc14b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 -- 172.21.15.106:0/833536671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc06f0c0 msgr2=0x7fd6bc15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc06f0c0 0x7fd6bc15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.473 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 -- 172.21.15.106:0/833536671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc0cd140 con 0x7fd6bc074aa0 2024-05-27T06:44:39.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c1000640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:44:39.474 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b3e00640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c3c0 tx=0x7fd6a400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4017070 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a40052f0 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4003860 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc0cbbd0 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc15a730 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6bc0cb9f0 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd6a401a070 con 0x7fd6bc074aa0 2024-05-27T06:44:39.475 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.482+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fd6a4011c00 con 0x7fd6bc074aa0 2024-05-27T06:44:39.477 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.486+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd6bc0cb9f0 con 0x7fd6bc074aa0 2024-05-27T06:44:39.482 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:39.490+0000 7fa2a8e28180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-27T06:44:39.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.554+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "43a3497e-e0ac-484e-afb6-23dcb6fd48d4", "id": 1} v 0) -- 0x7fd6bc06a560 con 0x7fd6bc074aa0 2024-05-27T06:44:39.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.558+0000 7fd6b8c00640 1 -- 172.21.15.106:0/833536671 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "43a3497e-e0ac-484e-afb6-23dcb6fd48d4", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fd6a4021070 con 0x7fd6bc074aa0 2024-05-27T06:44:39.551 INFO:teuthology.orchestra.run.smithi106.stdout:1 2024-05-27T06:44:39.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 msgr2=0x7fd6bc1e7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c3c0 tx=0x7fd6a400c880 comp rx=0 tx=0).stop 2024-05-27T06:44:39.552 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 shutdown_connections 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6bc15fe30 0x7fd6bc14b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6bc074aa0 0x7fd6bc1e7270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 --2- 172.21.15.106:0/833536671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6bc06f0c0 0x7fd6bc15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 >> 172.21.15.106:0/833536671 conn(0x7fd6bc072500 msgr2=0x7fd6bc14f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 shutdown_connections 2024-05-27T06:44:39.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.562+0000 7fd6c3000640 1 -- 172.21.15.106:0/833536671 wait complete. 2024-05-27T06:44:39.570 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb 2 2024-05-27T06:44:39.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 Processor -- start 2024-05-27T06:44:39.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 -- start start 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 0x7f44e80c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f44e80c5d30 0x7f44e815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e80752a0 con 0x7f44e8157b20 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f44e80c5560 con 0x7f44e8157f30 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ef000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f44e80c6140 con 0x7f44e80c5d30 2024-05-27T06:44:39.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f44e80c5d30 0x7f44e815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f44e80c5d30 0x7f44e815df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47370/0 (socket says 172.21.15.106:47370) 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 -- 172.21.15.106:0/4124930385 learned_addr learned my addr 172.21.15.106:0/4124930385 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44df400640 1 --2- 172.21.15.106:0/4124930385 >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 0x7f44e80c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 -- 172.21.15.106:0/4124930385 >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 msgr2=0x7f44e80c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 --2- 172.21.15.106:0/4124930385 >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 0x7f44e80c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 -- 172.21.15.106:0/4124930385 >> v1:172.21.15.106:6789/0 conn(0x7f44e8157b20 legacy=0x7f44e80c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.726+0000 7f44ed000640 1 -- 172.21.15.106:0/4124930385 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e80ca690 con 0x7f44e80c5d30 2024-05-27T06:44:39.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.730+0000 7f44df400640 1 --2- 172.21.15.106:0/4124930385 >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 0x7f44e80c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:39.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.730+0000 7f44ed000640 1 --2- 172.21.15.106:0/4124930385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e80c5d30 0x7f44e815df80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f44e806fcd0 tx=0x7f44e001f1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=816c9eaafe99aebb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.730+0000 7f44e7400640 1 -- 172.21.15.106:0/4124930385 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00b4070 con 0x7f44e80c5d30 2024-05-27T06:44:39.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.730+0000 7f44e7400640 1 -- 172.21.15.106:0/4124930385 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e00af400 con 0x7f44e80c5d30 2024-05-27T06:44:39.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.730+0000 7f44e7400640 1 -- 172.21.15.106:0/4124930385 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e00af7d0 con 0x7f44e80c5d30 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- 172.21.15.106:0/4124930385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e80c5d30 msgr2=0x7f44e815df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 --2- 172.21.15.106:0/4124930385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e80c5d30 0x7f44e815df80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f44e806fcd0 tx=0x7f44e001f1e0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- 172.21.15.106:0/4124930385 shutdown_connections 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 --2- 172.21.15.106:0/4124930385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e80c5d30 0x7f44e815df80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 --2- 172.21.15.106:0/4124930385 >> v2:172.21.15.81:3300/0 conn(0x7f44e8157f30 0x7f44e80c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- 172.21.15.106:0/4124930385 >> 172.21.15.106:0/4124930385 conn(0x7f44e8072500 msgr2=0x7f44e8072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- 172.21.15.106:0/4124930385 shutdown_connections 2024-05-27T06:44:39.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- 172.21.15.106:0/4124930385 wait complete. 2024-05-27T06:44:39.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 Processor -- start 2024-05-27T06:44:39.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.734+0000 7f44ef000640 1 -- start start 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 0x7f44e814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 0x7f44e80b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f44e80caf30 con 0x7f44e80c5d30 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f44e80cac50 con 0x7f44e8157b20 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f44e8074f30 con 0x7f44e8157f30 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47382/0 (socket says 172.21.15.106:47382) 2024-05-27T06:44:39.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 0x7f44e814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 -- 172.21.15.106:0/597749747 learned_addr learned my addr 172.21.15.106:0/597749747 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ed000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 0x7f44e80b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 -- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 msgr2=0x7f44e80b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 0x7f44e80b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 -- 172.21.15.106:0/597749747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 msgr2=0x7f44e814e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 0x7f44e814e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 -- 172.21.15.106:0/597749747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e8150ee0 con 0x7f44e8157b20 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ed000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 0x7f44e80b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e7e00640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 0x7f44e814e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44df400640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f44d000d860 tx=0x7f44d000dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d000de50 con 0x7f44e8157b20 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d0019070 con 0x7f44e8157b20 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e815d4e0 con 0x7f44e8157b20 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d000bb10 con 0x7f44e8157b20 2024-05-27T06:44:39.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e81510f0 con 0x7f44e8157b20 2024-05-27T06:44:39.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44d000bcb0 con 0x7f44e8157b20 2024-05-27T06:44:39.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f44d00148a0 con 0x7f44e8157b20 2024-05-27T06:44:39.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.738+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e806a560 con 0x7f44e8157b20 2024-05-27T06:44:39.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.742+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f44d0014480 con 0x7f44e8157b20 2024-05-27T06:44:39.801 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.810+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb", "id": 2} v 0) -- 0x7f44e81f13f0 con 0x7f44e8157b20 2024-05-27T06:44:39.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.814+0000 7f44e4c00640 1 -- 172.21.15.106:0/597749747 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f44d0014b80 con 0x7f44e8157b20 2024-05-27T06:44:39.807 INFO:teuthology.orchestra.run.smithi106.stdout:2 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 msgr2=0x7f44e81509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f44d000d860 tx=0x7f44d000dd20 comp rx=0 tx=0).stop 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 shutdown_connections 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44e8157f30 0x7f44e80b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44e8157b20 0x7f44e81509a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 --2- 172.21.15.106:0/597749747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c5d30 0x7f44e814e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 >> 172.21.15.106:0/597749747 conn(0x7f44e8072500 msgr2=0x7f44e80c4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 shutdown_connections 2024-05-27T06:44:39.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.818+0000 7f44ef000640 1 -- 172.21.15.106:0/597749747 wait complete. 2024-05-27T06:44:39.828 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0 3 2024-05-27T06:44:39.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.982+0000 7f7fb5800640 1 Processor -- start 2024-05-27T06:44:39.974 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.982+0000 7f7fb5800640 1 -- start start 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.982+0000 7f7fb5800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7fb0072c80 0x7f7fb006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.982+0000 7f7fb5800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7fb015a570 0x7f7fb00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fb5800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fb00ca3d0 con 0x7f7fb015a980 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fb5800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb0073c20 con 0x7f7fb015a570 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fb5800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7fb00ca6b0 con 0x7f7fb0072c80 2024-05-27T06:44:39.976 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7fb0072c80 0x7f7fb006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7fb0072c80 0x7f7fb006efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47398/0 (socket says 172.21.15.106:47398) 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 -- 172.21.15.106:0/4081499263 learned_addr learned my addr 172.21.15.106:0/4081499263 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fa7e00640 1 --2- 172.21.15.106:0/4081499263 >> v2:172.21.15.81:3300/0 conn(0x7f7fb015a570 0x7f7fb00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 -- 172.21.15.106:0/4081499263 >> v2:172.21.15.81:3300/0 conn(0x7f7fb015a570 msgr2=0x7f7fb00c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 --2- 172.21.15.106:0/4081499263 >> v2:172.21.15.81:3300/0 conn(0x7f7fb015a570 0x7f7fb00c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 -- 172.21.15.106:0/4081499263 >> v1:172.21.15.81:6789/0 conn(0x7f7fb015a980 legacy=0x7f7fb00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.977 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 -- 172.21.15.106:0/4081499263 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb0073e70 con 0x7f7fb0072c80 2024-05-27T06:44:39.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7faea00640 1 --2- 172.21.15.106:0/4081499263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb0072c80 0x7f7fb006efe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0073da0 tx=0x7f7fa001c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24a4ab2c5b31a79a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fae000640 1 -- 172.21.15.106:0/4081499263 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00b1070 con 0x7f7fb0072c80 2024-05-27T06:44:39.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fae000640 1 -- 172.21.15.106:0/4081499263 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa00ac600 con 0x7f7fb0072c80 2024-05-27T06:44:39.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.986+0000 7f7fae000640 1 -- 172.21.15.106:0/4081499263 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa00ac9d0 con 0x7f7fb0072c80 2024-05-27T06:44:39.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- 172.21.15.106:0/4081499263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb0072c80 msgr2=0x7f7fb006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 --2- 172.21.15.106:0/4081499263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb0072c80 0x7f7fb006efe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0073da0 tx=0x7f7fa001c490 comp rx=0 tx=0).stop 2024-05-27T06:44:39.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- 172.21.15.106:0/4081499263 shutdown_connections 2024-05-27T06:44:39.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 --2- 172.21.15.106:0/4081499263 >> v2:172.21.15.81:3300/0 conn(0x7f7fb015a570 0x7f7fb00c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 --2- 172.21.15.106:0/4081499263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb0072c80 0x7f7fb006efe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- 172.21.15.106:0/4081499263 >> 172.21.15.106:0/4081499263 conn(0x7f7fb0070820 msgr2=0x7f7fb0070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:39.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- 172.21.15.106:0/4081499263 shutdown_connections 2024-05-27T06:44:39.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- 172.21.15.106:0/4081499263 wait complete. 2024-05-27T06:44:39.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 Processor -- start 2024-05-27T06:44:39.983 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.990+0000 7f7fb5800640 1 -- start start 2024-05-27T06:44:39.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fb0072c80 0x7f7fb01f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.984 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7fb015a980 0x7f7fb01f9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb00c8b20 con 0x7f7fb0072c80 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb00c89a0 con 0x7f7fb015a570 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb0167f10 con 0x7f7fb015a980 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47402/0 (socket says 172.21.15.106:47402) 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 -- 172.21.15.106:0/743560023 learned_addr learned my addr 172.21.15.106:0/743560023 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7faf400640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7fb015a980 0x7f7fb01f9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.985 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7faea00640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fb0072c80 0x7f7fb01f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:39.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 -- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7fb015a980 msgr2=0x7f7fb01f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7fb015a980 0x7f7fb01f9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.986 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 -- 172.21.15.106:0/743560023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fb0072c80 msgr2=0x7f7fb01f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fb0072c80 0x7f7fb01f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 -- 172.21.15.106:0/743560023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb00cbee0 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa7e00640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e870 tx=0x7f7f9800ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f98017070 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f9800c940 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb01f73d0 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f9800ccf0 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb0150910 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f9801a070 con 0x7f7fb015a570 2024-05-27T06:44:39.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7f98006020 con 0x7f7fb015a570 2024-05-27T06:44:39.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.994+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb006a560 con 0x7f7fb015a570 2024-05-27T06:44:39.989 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:39.998+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7f98006480 con 0x7f7fb015a570 2024-05-27T06:44:40.056 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.066+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0", "id": 3} v 0) -- 0x7f7fb0050870 con 0x7f7fb015a570 2024-05-27T06:44:40.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.070+0000 7f7fa6a00640 1 -- 172.21.15.106:0/743560023 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7f98006740 con 0x7f7fb015a570 2024-05-27T06:44:40.063 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.072+0000 7fdcf602a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:44:40.063 INFO:teuthology.orchestra.run.smithi106.stdout:3 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 msgr2=0x7f7fb01478c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800e870 tx=0x7f7f9800ec70 comp rx=0 tx=0).stop 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 shutdown_connections 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7fb015a980 0x7f7fb01f9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7fb015a570 0x7f7fb01478c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 --2- 172.21.15.106:0/743560023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fb0072c80 0x7f7fb01f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 >> 172.21.15.106:0/743560023 conn(0x7f7fb0070820 msgr2=0x7f7fb0075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 shutdown_connections 2024-05-27T06:44:40.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.074+0000 7f7fb5800640 1 -- 172.21.15.106:0/743560023 wait complete. 2024-05-27T06:44:40.082 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 5ccb2aa7-99d1-408f-8922-501785026e51 4 2024-05-27T06:44:40.125 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.132+0000 7fdcf602a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:44:40.126 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.134+0000 7fa2a8e28180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-27T06:44:40.185 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.192+0000 7fdcf602a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:44:40.201 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.210+0000 7fa2a8e28180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-27T06:44:40.229 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 Processor -- start 2024-05-27T06:44:40.230 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 -- start start 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b2c157f30 0x7f3b2c0c57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 0x7f3b2c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2c0752a0 con 0x7f3b2c157b20 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2c0c5560 con 0x7f3b2c157f30 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.238+0000 7f3b32600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2c0c6140 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 0x7f3b2c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2b400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f3b2c157b20 0x7f3b2c0c4e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.106:33260/0 (socket says 172.21.15.106:33260) 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2b400640 1 -- 172.21.15.106:0/30327697 learned_addr learned my addr 172.21.15.106:0/30327697 (peer_addr_for_me v1:172.21.15.106:0/0) 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b23e00640 1 --2- 172.21.15.106:0/30327697 >> v2:172.21.15.81:3300/0 conn(0x7f3b2c157f30 0x7f3b2c0c57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 -- 172.21.15.106:0/30327697 >> v2:172.21.15.81:3300/0 conn(0x7f3b2c157f30 msgr2=0x7f3b2c0c57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 --2- 172.21.15.106:0/30327697 >> v2:172.21.15.81:3300/0 conn(0x7f3b2c157f30 0x7f3b2c0c57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 -- 172.21.15.106:0/30327697 >> v1:172.21.15.106:6789/0 conn(0x7f3b2c157b20 legacy=0x7f3b2c0c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2aa00640 1 -- 172.21.15.106:0/30327697 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731950014 0 0) 0x7f3b2c0752a0 con 0x7f3b2c157b20 2024-05-27T06:44:40.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 -- 172.21.15.106:0/30327697 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c0ca690 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2be00640 1 --2- 172.21.15.106:0/30327697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 0x7f3b2c15df80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009d90 tx=0x7f3b2401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=716694da45d4933a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2aa00640 1 -- 172.21.15.106:0/30327697 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240b4070 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2aa00640 1 -- 172.21.15.106:0/30327697 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b240af600 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.242+0000 7f3b2aa00640 1 -- 172.21.15.106:0/30327697 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240af9d0 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 -- 172.21.15.106:0/30327697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 msgr2=0x7f3b2c15df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 --2- 172.21.15.106:0/30327697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 0x7f3b2c15df80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009d90 tx=0x7f3b2401f150 comp rx=0 tx=0).stop 2024-05-27T06:44:40.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 -- 172.21.15.106:0/30327697 shutdown_connections 2024-05-27T06:44:40.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 --2- 172.21.15.106:0/30327697 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c0c5d30 0x7f3b2c15df80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.236 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 --2- 172.21.15.106:0/30327697 >> v2:172.21.15.81:3300/0 conn(0x7f3b2c157f30 0x7f3b2c0c57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 -- 172.21.15.106:0/30327697 >> 172.21.15.106:0/30327697 conn(0x7f3b2c072500 msgr2=0x7f3b2c072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 -- 172.21.15.106:0/30327697 shutdown_connections 2024-05-27T06:44:40.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.246+0000 7f3b32600640 1 -- 172.21.15.106:0/30327697 wait complete. 2024-05-27T06:44:40.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 Processor -- start 2024-05-27T06:44:40.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 -- start start 2024-05-27T06:44:40.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 0x7f3b2c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 0x7f3b2c149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c0caf30 con 0x7f3b2c0c5d30 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c0cac50 con 0x7f3b2c157b20 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b32600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2c074f30 con 0x7f3b2c157f30 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b23e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 0x7f3b2c149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 0x7f3b2c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b23e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 0x7f3b2c149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47416/0 (socket says 172.21.15.106:47416) 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44438/0 (socket says 172.21.15.106:44438) 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b23e00640 1 -- 172.21.15.106:0/3665894446 learned_addr learned my addr 172.21.15.106:0/3665894446 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 0x7f3b2c0cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.106:60036/0 (socket says 172.21.15.106:60036) 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 -- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 msgr2=0x7f3b2c149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 0x7f3b2c149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 -- 172.21.15.106:0/3665894446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 msgr2=0x7f3b2c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 0x7f3b2c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.250+0000 7f3b2be00640 1 -- 172.21.15.106:0/3665894446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c1e52b0 con 0x7f3b2c157f30 2024-05-27T06:44:40.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b2be00640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3b240af000 tx=0x7f3b2401bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240bf030 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b240af400 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c0ce3f0 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240b4040 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c1e6a50 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3b240c2070 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f3b240af5b0 con 0x7f3b2c157f30 2024-05-27T06:44:40.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.254+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c06a560 con 0x7f3b2c157f30 2024-05-27T06:44:40.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.258+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f3b240ca020 con 0x7f3b2c157f30 2024-05-27T06:44:40.268 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.278+0000 7fa2a8e28180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-27T06:44:40.274 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.284+0000 7fdcf602a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:44:40.314 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.322+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5ccb2aa7-99d1-408f-8922-501785026e51", "id": 4} v 0) -- 0x7f3b2c06d220 con 0x7f3b2c157f30 2024-05-27T06:44:40.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b23400640 1 -- 172.21.15.106:0/3665894446 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ccb2aa7-99d1-408f-8922-501785026e51", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f3b24018cb0 con 0x7f3b2c157f30 2024-05-27T06:44:40.321 INFO:teuthology.orchestra.run.smithi106.stdout:4 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 msgr2=0x7f3b2c1e74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3b240af000 tx=0x7f3b2401bef0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 shutdown_connections 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2c157f30 0x7f3b2c1e74b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b2c157b20 0x7f3b2c149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 --2- 172.21.15.106:0/3665894446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b2c0c5d30 0x7f3b2c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 >> 172.21.15.106:0/3665894446 conn(0x7f3b2c072500 msgr2=0x7f3b2c072910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 shutdown_connections 2024-05-27T06:44:40.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.330+0000 7f3b32600640 1 -- 172.21.15.106:0/3665894446 wait complete. 2024-05-27T06:44:40.336 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.344+0000 7fdcf602a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:44:40.342 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new 74250015-c353-4668-a84c-2adf06bf7bf3 5 2024-05-27T06:44:40.362 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.370+0000 7fa2a8e28180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-27T06:44:40.424 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.432+0000 7fdcf602a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:44:40.425 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.434+0000 7fa2a8e28180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-27T06:44:40.487 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.496+0000 7fdcf602a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:44:40.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 Processor -- start 2024-05-27T06:44:40.512 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 -- start start 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f308c0b5da0 0x7f308c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f308c1608f0 0x7f308c0c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f308c0c7720 con 0x7f308c073ec0 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f308c160d00 con 0x7f308c0b5da0 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3092000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f308c0c8310 con 0x7f308c1608f0 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f308c1608f0 0x7f308c0c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f308c1608f0 0x7f308c0c87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47430/0 (socket says 172.21.15.106:47430) 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 -- 172.21.15.106:0/3517438637 learned_addr learned my addr 172.21.15.106:0/3517438637 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f308b400640 1 --2- 172.21.15.106:0/3517438637 >> v2:172.21.15.81:3300/0 conn(0x7f308c0b5da0 0x7f308c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.515 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 -- 172.21.15.106:0/3517438637 >> v2:172.21.15.81:3300/0 conn(0x7f308c0b5da0 msgr2=0x7f308c0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:40.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 --2- 172.21.15.106:0/3517438637 >> v2:172.21.15.81:3300/0 conn(0x7f308c0b5da0 0x7f308c0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 -- 172.21.15.106:0/3517438637 >> v1:172.21.15.81:6789/0 conn(0x7f308c073ec0 legacy=0x7f308c15c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.516 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.522+0000 7f3082a00640 1 -- 172.21.15.106:0/3517438637 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c0c79e0 con 0x7f308c1608f0 2024-05-27T06:44:40.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3082a00640 1 --2- 172.21.15.106:0/3517438637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c1608f0 0x7f308c0c87a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f308c15fe70 tx=0x7f307401ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33ff6eecbd9a1025 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f308aa00640 1 -- 172.21.15.106:0/3517438637 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30740b5070 con 0x7f308c1608f0 2024-05-27T06:44:40.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f308aa00640 1 -- 172.21.15.106:0/3517438637 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30740a8a10 con 0x7f308c1608f0 2024-05-27T06:44:40.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f308aa00640 1 -- 172.21.15.106:0/3517438637 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30740a8de0 con 0x7f308c1608f0 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 -- 172.21.15.106:0/3517438637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c1608f0 msgr2=0x7f308c0c87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 --2- 172.21.15.106:0/3517438637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c1608f0 0x7f308c0c87a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f308c15fe70 tx=0x7f307401ed10 comp rx=0 tx=0).stop 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 -- 172.21.15.106:0/3517438637 shutdown_connections 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 --2- 172.21.15.106:0/3517438637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c1608f0 0x7f308c0c87a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 --2- 172.21.15.106:0/3517438637 >> v2:172.21.15.81:3300/0 conn(0x7f308c0b5da0 0x7f308c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 -- 172.21.15.106:0/3517438637 >> 172.21.15.106:0/3517438637 conn(0x7f308c06ed00 msgr2=0x7f308c0d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 -- 172.21.15.106:0/3517438637 shutdown_connections 2024-05-27T06:44:40.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.526+0000 7f3092000640 1 -- 172.21.15.106:0/3517438637 wait complete. 2024-05-27T06:44:40.520 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.530+0000 7fa2a8e28180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-27T06:44:40.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 Processor -- start 2024-05-27T06:44:40.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 -- start start 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f308c1608f0 0x7f308c14eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f308c0c8ce0 con 0x7f308c1608f0 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f308c0c7fa0 con 0x7f308c073ec0 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3092000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f308c0b48f0 con 0x7f308c0b5da0 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3082a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47440/0 (socket says 172.21.15.106:47440) 2024-05-27T06:44:40.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 -- 172.21.15.106:0/1442991443 learned_addr learned my addr 172.21.15.106:0/1442991443 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:40.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3082a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44452/0 (socket says 172.21.15.106:44452) 2024-05-27T06:44:40.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308be00640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f308c1608f0 0x7f308c14eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 -- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 msgr2=0x7f308c1599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 -- 172.21.15.106:0/1442991443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f308c1608f0 msgr2=0x7f308c14eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f308c1608f0 0x7f308c14eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f308b400640 1 -- 172.21.15.106:0/1442991443 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c1e71e0 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.530+0000 7f3082a00640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f308b400640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f307c00e870 tx=0x7f307c00ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c017070 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307c00c940 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308c1f3ed0 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c00ccf0 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f308c1f7160 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f307c01a070 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f307c0139f0 con 0x7f308c073ec0 2024-05-27T06:44:40.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.534+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c1e73b0 con 0x7f308c073ec0 2024-05-27T06:44:40.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.546+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f307c013470 con 0x7f308c073ec0 2024-05-27T06:44:40.585 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.594+0000 7fa2a8e28180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-27T06:44:40.602 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.610+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "74250015-c353-4668-a84c-2adf06bf7bf3", "id": 5} v 0) -- 0x7f308c06a560 con 0x7f308c073ec0 2024-05-27T06:44:40.609 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3083e00640 1 -- 172.21.15.106:0/1442991443 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74250015-c353-4668-a84c-2adf06bf7bf3", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f307c013650 con 0x7f308c073ec0 2024-05-27T06:44:40.609 INFO:teuthology.orchestra.run.smithi106.stdout:5 2024-05-27T06:44:40.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 msgr2=0x7f308c1f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f307c00e870 tx=0x7f307c00ec70 comp rx=0 tx=0).stop 2024-05-27T06:44:40.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 shutdown_connections 2024-05-27T06:44:40.610 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f308c0b5da0 0x7f308c1599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f308c1608f0 0x7f308c14eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 --2- 172.21.15.106:0/1442991443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f308c073ec0 0x7f308c1f3990 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 >> 172.21.15.106:0/1442991443 conn(0x7f308c06ed00 msgr2=0x7f308c155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 shutdown_connections 2024-05-27T06:44:40.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.618+0000 7f3092000640 1 -- 172.21.15.106:0/1442991443 wait complete. 2024-05-27T06:44:40.629 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new afac5d3f-cbc1-4496-9f0f-833d2974d622 6 2024-05-27T06:44:40.654 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.664+0000 7fdcf602a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:44:40.725 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:40.732+0000 7fdcf602a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:44:40.759 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.766+0000 7fa2a8e28180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-27T06:44:40.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 Processor -- start 2024-05-27T06:44:40.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 -- start start 2024-05-27T06:44:40.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7604074ac0 0x7f7604074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f760406ea60 0x7f760406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f760406dff0 con 0x7f760406ea60 2024-05-27T06:44:40.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7604160600 con 0x7f7604074ac0 2024-05-27T06:44:40.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.786+0000 7f7609c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76041608e0 con 0x7f760415fe30 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7604074ac0 0x7f7604074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44454/0 (socket says 172.21.15.106:44454) 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602a00640 1 -- 172.21.15.106:0/1673705667 learned_addr learned my addr 172.21.15.106:0/1673705667 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7603400640 1 --2- 172.21.15.106:0/1673705667 >> v2:172.21.15.81:3300/0 conn(0x7f760406ea60 0x7f760406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 -- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 msgr2=0x7f76040ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 -- 172.21.15.106:0/1673705667 >> v2:172.21.15.81:3300/0 conn(0x7f760406ea60 msgr2=0x7f760406ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 --2- 172.21.15.106:0/1673705667 >> v2:172.21.15.81:3300/0 conn(0x7f760406ea60 0x7f760406ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602000640 1 -- 172.21.15.106:0/1673705667 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7604160bc0 con 0x7f7604074ac0 2024-05-27T06:44:40.781 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.790+0000 7f7602a00640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:44:40.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7602000640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7604074ac0 0x7f7604074ed0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f76040755f0 tx=0x7f75f001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba25be241f38702f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7601600640 1 -- 172.21.15.106:0/1673705667 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00b0070 con 0x7f7604074ac0 2024-05-27T06:44:40.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7601600640 1 -- 172.21.15.106:0/1673705667 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f00ab600 con 0x7f7604074ac0 2024-05-27T06:44:40.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7601600640 1 -- 172.21.15.106:0/1673705667 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00ab9d0 con 0x7f7604074ac0 2024-05-27T06:44:40.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 -- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7604074ac0 msgr2=0x7f7604074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7604074ac0 0x7f7604074ed0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f76040755f0 tx=0x7f75f001c480 comp rx=0 tx=0).stop 2024-05-27T06:44:40.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 -- 172.21.15.106:0/1673705667 shutdown_connections 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 --2- 172.21.15.106:0/1673705667 >> v2:172.21.15.81:3300/0 conn(0x7f760406ea60 0x7f760406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7604074ac0 0x7f7604074ed0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 --2- 172.21.15.106:0/1673705667 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760415fe30 0x7f76040ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 -- 172.21.15.106:0/1673705667 >> 172.21.15.106:0/1673705667 conn(0x7f7604072500 msgr2=0x7f7604072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 -- 172.21.15.106:0/1673705667 shutdown_connections 2024-05-27T06:44:40.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.794+0000 7f7609c00640 1 -- 172.21.15.106:0/1673705667 wait complete. 2024-05-27T06:44:40.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 Processor -- start 2024-05-27T06:44:40.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 -- start start 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7604074ac0 0x7f76040cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f760415fe30 0x7f760414a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76041603c0 con 0x7f7604074ac0 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7604160240 con 0x7f760415fe30 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7609c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76040caa30 con 0x7f760406ea60 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44470/0 (socket says 172.21.15.106:44470) 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7603400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f760415fe30 0x7f760414a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.791 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 -- 172.21.15.106:0/2661926371 learned_addr learned my addr 172.21.15.106:0/2661926371 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602000640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7604074ac0 0x7f76040cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 -- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f760415fe30 msgr2=0x7f760414a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f760415fe30 0x7f760414a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 -- 172.21.15.106:0/2661926371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7604074ac0 msgr2=0x7f76040cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7604074ac0 0x7f76040cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.792 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 -- 172.21.15.106:0/2661926371 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76041f6b00 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.798+0000 7f7602a00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f75f400e930 tx=0x7f75f400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4017070 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f400ca00 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760415eae0 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f400cdd0 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f760415e2f0 con 0x7f760406ea60 2024-05-27T06:44:40.793 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75f401a070 con 0x7f760406ea60 2024-05-27T06:44:40.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f75f4006020 con 0x7f760406ea60 2024-05-27T06:44:40.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f760406a560 con 0x7f760406ea60 2024-05-27T06:44:40.795 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.802+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f75f40064d0 con 0x7f760406ea60 2024-05-27T06:44:40.844 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:40.854+0000 7fa2a8e28180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-27T06:44:40.872 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.882+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd new", "uuid": "afac5d3f-cbc1-4496-9f0f-833d2974d622", "id": 6} v 0) -- 0x7f76041f61e0 con 0x7f760406ea60 2024-05-27T06:44:40.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f75faa00640 1 -- 172.21.15.106:0/2661926371 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "afac5d3f-cbc1-4496-9f0f-833d2974d622", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f75f40066b0 con 0x7f760406ea60 2024-05-27T06:44:40.878 INFO:teuthology.orchestra.run.smithi106.stdout:6 2024-05-27T06:44:40.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 msgr2=0x7f76041f65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f75f400e930 tx=0x7f75f400edf0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 shutdown_connections 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f760415fe30 0x7f760414a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7604074ac0 0x7f76040cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 --2- 172.21.15.106:0/2661926371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f760406ea60 0x7f76041f65c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 >> 172.21.15.106:0/2661926371 conn(0x7f7604072500 msgr2=0x7f760414e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 shutdown_connections 2024-05-27T06:44:40.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:40.886+0000 7f7609c00640 1 -- 172.21.15.106:0/2661926371 wait complete. 2024-05-27T06:44:40.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo ceph --cluster ceph osd new afafd7c5-606b-4724-acbc-f386c4f729b7 7 2024-05-27T06:44:41.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 Processor -- start 2024-05-27T06:44:41.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 -- start start 2024-05-27T06:44:41.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 0x7f9834073e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98340c67f0 con 0x7f98340c6bd0 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9834072450 con 0x7f983415a230 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9838e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f98340c53f0 con 0x7f98341595d0 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 0x7f9834073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 0x7f9834073e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47460/0 (socket says 172.21.15.106:47460) 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.106:44478/0 (socket says 172.21.15.106:44478) 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831400640 1 -- 172.21.15.106:0/2993202889 learned_addr learned my addr 172.21.15.106:0/2993202889 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 -- 172.21.15.106:0/2993202889 >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 msgr2=0x7f9834073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 --2- 172.21.15.106:0/2993202889 >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 0x7f9834073e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 -- 172.21.15.106:0/2993202889 >> v1:172.21.15.106:6789/0 conn(0x7f98340c6bd0 legacy=0x7f98340743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.058+0000 7f9831e00640 1 -- 172.21.15.106:0/2993202889 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98340726a0 con 0x7f98341595d0 2024-05-27T06:44:41.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9831e00640 1 --2- 172.21.15.106:0/2993202889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f98240097c0 tx=0x7f982401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff84a0bfba9c8fbc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:41.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9830a00640 1 -- 172.21.15.106:0/2993202889 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240aa620 con 0x7f98341595d0 2024-05-27T06:44:41.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9830a00640 1 -- 172.21.15.106:0/2993202889 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98240af070 con 0x7f98341595d0 2024-05-27T06:44:41.054 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9830a00640 1 -- 172.21.15.106:0/2993202889 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240aa9e0 con 0x7f98341595d0 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 -- 172.21.15.106:0/2993202889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 msgr2=0x7f98340c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 --2- 172.21.15.106:0/2993202889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f98240097c0 tx=0x7f982401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 -- 172.21.15.106:0/2993202889 shutdown_connections 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 --2- 172.21.15.106:0/2993202889 >> v2:172.21.15.106:3300/0 conn(0x7f983415a230 0x7f9834073e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 --2- 172.21.15.106:0/2993202889 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f98341595d0 0x7f98340c77e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 -- 172.21.15.106:0/2993202889 >> 172.21.15.106:0/2993202889 conn(0x7f9834070760 msgr2=0x7f98340cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 -- 172.21.15.106:0/2993202889 shutdown_connections 2024-05-27T06:44:41.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.062+0000 7f9838e00640 1 -- 172.21.15.106:0/2993202889 wait complete. 2024-05-27T06:44:41.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 Processor -- start 2024-05-27T06:44:41.057 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 -- start start 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 0x7f9834152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f983415a230 0x7f983414c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9834164680 con 0x7f98340c6bd0 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9834072980 con 0x7f98341595d0 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9838e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9834164800 con 0x7f983415a230 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9831400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9831400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47476/0 (socket says 172.21.15.106:47476) 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9832800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f983415a230 0x7f983414c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9831400640 1 -- 172.21.15.106:0/533143148 learned_addr learned my addr 172.21.15.106:0/533143148 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-05-27T06:44:41.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.066+0000 7f9831e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 0x7f9834152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 -- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f983415a230 msgr2=0x7f983414c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f983415a230 0x7f983414c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 -- 172.21.15.106:0/533143148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 msgr2=0x7f9834152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 0x7f9834152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 -- 172.21.15.106:0/533143148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98340cc740 con 0x7f98341595d0 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 0x7f9834152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:44:41.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9831400640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f982000e870 tx=0x7f982000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820017070 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982000ee60 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982000cae0 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98341532c0 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98341528a0 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f982001a070 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f98200059a0 con 0x7f98341595d0 2024-05-27T06:44:41.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f983406a560 con 0x7f98341595d0 2024-05-27T06:44:41.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.070+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f982000cce0 con 0x7f98341595d0 2024-05-27T06:44:41.129 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.138+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd new", "uuid": "afafd7c5-606b-4724-acbc-f386c4f729b7", "id": 7} v 0) -- 0x7f9834150e50 con 0x7f98341595d0 2024-05-27T06:44:41.136 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f982a000640 1 -- 172.21.15.106:0/533143148 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "afafd7c5-606b-4724-acbc-f386c4f729b7", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f9820003620 con 0x7f98341595d0 2024-05-27T06:44:41.136 INFO:teuthology.orchestra.run.smithi106.stdout:7 2024-05-27T06:44:41.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 msgr2=0x7f9834143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f982000e870 tx=0x7f982000ed30 comp rx=0 tx=0).stop 2024-05-27T06:44:41.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 shutdown_connections 2024-05-27T06:44:41.137 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f983415a230 0x7f983414c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98341595d0 0x7f9834143570 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 --2- 172.21.15.106:0/533143148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98340c6bd0 0x7f9834152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 >> 172.21.15.106:0/533143148 conn(0x7f9834070760 msgr2=0x7f9834147450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:41.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 shutdown_connections 2024-05-27T06:44:41.138 INFO:teuthology.orchestra.run.smithi106.stderr:2024-05-27T06:44:41.146+0000 7f9838e00640 1 -- 172.21.15.106:0/533143148 wait complete. 2024-05-27T06:44:41.161 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-27T06:44:41.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-27T06:44:41.164 INFO:tasks.ceph.osd.0:Started 2024-05-27T06:44:41.164 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-27T06:44:41.164 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-27T06:44:41.167 INFO:tasks.ceph.osd.1:Started 2024-05-27T06:44:41.167 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-27T06:44:41.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-27T06:44:41.170 INFO:tasks.ceph.osd.2:Started 2024-05-27T06:44:41.170 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-27T06:44:41.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-27T06:44:41.174 INFO:tasks.ceph.osd.3:Started 2024-05-27T06:44:41.174 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-27T06:44:41.174 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-27T06:44:41.178 INFO:tasks.ceph.osd.4:Started 2024-05-27T06:44:41.179 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-27T06:44:41.179 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-27T06:44:41.188 INFO:tasks.ceph.osd.5:Started 2024-05-27T06:44:41.188 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-27T06:44:41.188 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-27T06:44:41.192 INFO:tasks.ceph.osd.6:Started 2024-05-27T06:44:41.192 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-27T06:44:41.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-27T06:44:41.203 INFO:tasks.ceph.osd.7:Started 2024-05-27T06:44:41.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:44:41.335 INFO:tasks.ceph.mgr.y.smithi081.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-27T06:44:41.336 INFO:tasks.ceph.mgr.y.smithi081.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-27T06:44:41.336 INFO:tasks.ceph.mgr.y.smithi081.stderr: from numpy import show_config as show_numpy_config 2024-05-27T06:44:41.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 Processor -- start 2024-05-27T06:44:41.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 -- start start 2024-05-27T06:44:41.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd5b0160880 0x7fd5b01680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b00cd390 con 0x7fd5b0160470 2024-05-27T06:44:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b006f120 con 0x7fd5b0156900 2024-05-27T06:44:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b7c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b015fd00 con 0x7fd5b0160880 2024-05-27T06:44:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5a7e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd5b0160470 0x7fd5b006c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52102/0 (socket says 172.21.15.81:52102) 2024-05-27T06:44:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5a7e00640 1 -- 172.21.15.81:0/2983467789 learned_addr learned my addr 172.21.15.81:0/2983467789 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:44:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b5c00640 1 --2- 172.21.15.81:0/2983467789 >> v2:172.21.15.106:3300/0 conn(0x7fd5b0160880 0x7fd5b01680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780765916 0 0) 0x7fd5b00cd390 con 0x7fd5b0160470 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd594003530 con 0x7fd5b0160470 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780913061 0 0) 0x7fd5b006f120 con 0x7fd5b0156900 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5b00cd390 con 0x7fd5b0156900 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b5c00640 1 -- 172.21.15.81:0/2983467789 >> v1:172.21.15.106:6789/0 conn(0x7fd5b0156900 legacy=0x7fd5b00cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b5c00640 1 -- 172.21.15.81:0/2983467789 >> v1:172.21.15.81:6789/0 conn(0x7fd5b0160470 legacy=0x7fd5b006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.403+0000 7fd5b5c00640 1 -- 172.21.15.81:0/2983467789 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b006f400 con 0x7fd5b0160880 2024-05-27T06:44:41.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b5c00640 1 --2- 172.21.15.81:0/2983467789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0160880 0x7fd5b01680a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd5b015fe80 tx=0x7fd5a801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=684629db8bb51f27 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:41.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80b4070 con 0x7fd5b0160880 2024-05-27T06:44:41.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a80af600 con 0x7fd5b0160880 2024-05-27T06:44:41.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5afe00640 1 -- 172.21.15.81:0/2983467789 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80af9d0 con 0x7fd5b0160880 2024-05-27T06:44:41.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 -- 172.21.15.81:0/2983467789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0160880 msgr2=0x7fd5b01680a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/2983467789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0160880 0x7fd5b01680a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd5b015fe80 tx=0x7fd5a801f130 comp rx=0 tx=0).stop 2024-05-27T06:44:41.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 -- 172.21.15.81:0/2983467789 shutdown_connections 2024-05-27T06:44:41.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/2983467789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0160880 0x7fd5b01680a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 -- 172.21.15.81:0/2983467789 >> 172.21.15.81:0/2983467789 conn(0x7fd5b00cb2e0 msgr2=0x7fd5b00cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:41.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 -- 172.21.15.81:0/2983467789 shutdown_connections 2024-05-27T06:44:41.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.407+0000 7fd5b7c00640 1 -- 172.21.15.81:0/2983467789 wait complete. 2024-05-27T06:44:41.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 Processor -- start 2024-05-27T06:44:41.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 -- start start 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0156900 0x7fd5b0158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 0x7fd5b01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b006f9c0 con 0x7fd5b0160470 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b015ffe0 con 0x7fd5b0156900 2024-05-27T06:44:41.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b7c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b006d5b0 con 0x7fd5b0160880 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44292/0 (socket says 172.21.15.81:44292) 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5a7e00640 1 -- 172.21.15.81:0/3840104742 learned_addr learned my addr 172.21.15.81:0/3840104742 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b5200640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0156900 0x7fd5b0158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.411+0000 7fd5b5c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 0x7fd5b01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 -- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 msgr2=0x7fd5b01591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 0x7fd5b01591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 -- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0156900 msgr2=0x7fd5b0158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0156900 0x7fd5b0158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 -- 172.21.15.81:0/3840104742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b01587a0 con 0x7fd5b0160470 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5b5c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 0x7fd5b01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5a7e00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5980013b0 tx=0x7fd59800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:41.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598017070 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5980074e0 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980059f0 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b015f990 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b0074530 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd59801a070 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fd598005b90 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b006a560 con 0x7fd5b0160470 2024-05-27T06:44:41.409 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:41.415+0000 7fdcf602a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:44:41.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.419+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd598021020 con 0x7fd5b0160470 2024-05-27T06:44:41.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5b00ccf40 con 0x7fd5b0160470 2024-05-27T06:44:41.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5ad600640 1 -- 172.21.15.81:0/3840104742 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fd59801c440 con 0x7fd5b0160470 2024-05-27T06:44:41.468 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:41.468 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:44:41.140843+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 msgr2=0x7fd5b01e8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5980013b0 tx=0x7fd59800c890 comp rx=0 tx=0).stop 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 shutdown_connections 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5b0160880 0x7fd5b01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b0160470 0x7fd5b01e8150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 --2- 172.21.15.81:0/3840104742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5b0156900 0x7fd5b0158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 >> 172.21.15.81:0/3840104742 conn(0x7fd5b00cb2e0 msgr2=0x7fd5b006ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 shutdown_connections 2024-05-27T06:44:41.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:41.475+0000 7fd5b7c00640 1 -- 172.21.15.81:0/3840104742 wait complete. 2024-05-27T06:44:41.489 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-27T06:44:41.489 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-27T06:44:41.560 INFO:tasks.ceph.mgr.x.smithi106.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-27T06:44:41.560 INFO:tasks.ceph.mgr.x.smithi106.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-27T06:44:41.560 INFO:tasks.ceph.mgr.x.smithi106.stderr: from numpy import show_config as show_numpy_config 2024-05-27T06:44:41.564 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:41.571+0000 7fdcf602a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:44:41.677 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:41.685+0000 7fa2a8e28180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-27T06:44:41.742 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:41.751+0000 7fdcf602a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:44:41.815 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:41.823+0000 7fdcf602a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:44:41.825 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:41.833+0000 7fa2a8e28180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:44:41.848 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T06:44:41.855+0000 7efce0f387c0 -1 Falling back to public interface 2024-05-27T06:44:41.859 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T06:44:41.867+0000 7f5d9fc937c0 -1 Falling back to public interface 2024-05-27T06:44:41.866 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T06:44:41.873+0000 7fe646d437c0 -1 Falling back to public interface 2024-05-27T06:44:41.866 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T06:44:41.873+0000 7f1af00127c0 -1 Falling back to public interface 2024-05-27T06:44:41.867 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T06:44:41.875+0000 7fdff035a7c0 -1 Falling back to public interface 2024-05-27T06:44:41.871 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T06:44:41.879+0000 7fc08c02c7c0 -1 Falling back to public interface 2024-05-27T06:44:41.881 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T06:44:41.889+0000 7fce083827c0 -1 Falling back to public interface 2024-05-27T06:44:41.889 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T06:44:41.897+0000 7f1337c397c0 -1 Falling back to public interface 2024-05-27T06:44:41.973 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:41.979+0000 7fdcf602a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:44:42.005 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:42.013+0000 7fa2a8e28180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-27T06:44:42.098 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:42.105+0000 7fa2a8e28180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-27T06:44:42.250 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:42.257+0000 7fa2a8e28180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-27T06:44:42.295 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T06:44:42.434 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 Processor -- start 2024-05-27T06:44:42.434 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 -- start start 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770815d400 0x7f77080c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77080cae60 0x7f7708160740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7708160c80 con 0x7f770806e290 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f77080ca480 con 0x7f77080cae60 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7710400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f77080ca720 con 0x7f770815d400 2024-05-27T06:44:42.436 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77080cae60 0x7f7708160740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77080cae60 0x7f7708160740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44300/0 (socket says 172.21.15.81:44300) 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 -- 172.21.15.81:0/1732188529 learned_addr learned my addr 172.21.15.81:0/1732188529 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770da00640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770815d400 0x7f77080c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 -- 172.21.15.81:0/1732188529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770815d400 msgr2=0x7f77080c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770815d400 0x7f77080c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 -- 172.21.15.81:0/1732188529 >> v1:172.21.15.106:6789/0 conn(0x7f770806e290 legacy=0x7f7708071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:42.437 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f770d000640 1 -- 172.21.15.81:0/1732188529 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77080caa00 con 0x7f77080cae60 2024-05-27T06:44:42.438 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.443+0000 7f7707e00640 1 -- 172.21.15.81:0/1732188529 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766053936 0 0) 0x7f7708160c80 con 0x7f770806e290 2024-05-27T06:44:42.439 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f770d000640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f7708160740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2295a0e893114a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:42.440 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7707e00640 1 -- 172.21.15.81:0/1732188529 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f40b5070 con 0x7f77080cae60 2024-05-27T06:44:42.440 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7707e00640 1 -- 172.21.15.81:0/1732188529 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f40a8a30 con 0x7f77080cae60 2024-05-27T06:44:42.440 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7707e00640 1 -- 172.21.15.81:0/1732188529 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f40a8e00 con 0x7f77080cae60 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 -- 172.21.15.81:0/1732188529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 msgr2=0x7f7708160740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f7708160740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f76f4006fd0 tx=0x7f76f401ed30 comp rx=0 tx=0).stop 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 -- 172.21.15.81:0/1732188529 shutdown_connections 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f7708160740 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 --2- 172.21.15.81:0/1732188529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770815d400 0x7f77080c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 -- 172.21.15.81:0/1732188529 >> 172.21.15.81:0/1732188529 conn(0x7f77080763e0 msgr2=0x7f7708074480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 -- 172.21.15.81:0/1732188529 shutdown_connections 2024-05-27T06:44:42.441 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.447+0000 7f7710400640 1 -- 172.21.15.81:0/1732188529 wait complete. 2024-05-27T06:44:42.443 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 Processor -- start 2024-05-27T06:44:42.443 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 -- start start 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770806e290 0x7f77081f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 0x7f770814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f770815cee0 con 0x7f77080cae60 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f770815cc00 con 0x7f770815d400 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f77080cb270 con 0x7f770806e290 2024-05-27T06:44:42.446 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44302/0 (socket says 172.21.15.81:44302) 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 -- 172.21.15.81:0/51440239 learned_addr learned my addr 172.21.15.81:0/51440239 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770da00640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770806e290 0x7f77081f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770e400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 0x7f770814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 -- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770806e290 msgr2=0x7f77081f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770806e290 0x7f77081f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 -- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 msgr2=0x7f770814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 0x7f770814d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 -- 172.21.15.81:0/51440239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77081f70f0 con 0x7f77080cae60 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770e400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 0x7f770814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f770d000640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76f401c960 tx=0x7f76f401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f40b5070 con 0x7f77080cae60 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f4003620 con 0x7f77080cae60 2024-05-27T06:44:42.447 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77081f7300 con 0x7f77080cae60 2024-05-27T06:44:42.448 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.451+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4003a10 con 0x7f77080cae60 2024-05-27T06:44:42.448 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.455+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77080d0340 con 0x7f77080cae60 2024-05-27T06:44:42.448 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.455+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f76f40c0060 con 0x7f77080cae60 2024-05-27T06:44:42.448 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.455+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f76f40b0c10 con 0x7f77080cae60 2024-05-27T06:44:42.448 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.455+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77080774a0 con 0x7f77080cae60 2024-05-27T06:44:42.449 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.455+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f76f4018890 con 0x7f77080cae60 2024-05-27T06:44:42.525 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.531+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f770806de60 con 0x7f77080cae60 2024-05-27T06:44:42.525 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.531+0000 7f7705600640 1 -- 172.21.15.81:0/51440239 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f76f40c8020 con 0x7f77080cae60 2024-05-27T06:44:42.526 INFO:teuthology.misc.health.smithi081.stdout: 2024-05-27T06:44:42.526 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:44:41.140843+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 msgr2=0x7f77080cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f76f401c960 tx=0x7f76f401bad0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 shutdown_connections 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f770815d400 0x7f770814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77080cae60 0x7f77080cb650 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 --2- 172.21.15.81:0/51440239 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f770806e290 0x7f77081f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 >> 172.21.15.81:0/51440239 conn(0x7f77080763e0 msgr2=0x7f77080c89f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 shutdown_connections 2024-05-27T06:44:42.527 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:42.535+0000 7f7710400640 1 -- 172.21.15.81:0/51440239 wait complete. 2024-05-27T06:44:42.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-27T06:44:42.684 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:42.691+0000 7fdcf602a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:44:42.771 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:42.779+0000 7fdcf602a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:44:42.974 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:42.981+0000 7fa2a8e28180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-27T06:44:42.976 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:42.983+0000 7fdcf602a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:44:43.060 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:43.067+0000 7fdcf602a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:44:43.062 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:43.069+0000 7fa2a8e28180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-27T06:44:43.266 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:43.273+0000 7fa2a8e28180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-27T06:44:43.351 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:43.357+0000 7fa2a8e28180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-27T06:44:43.392 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:43.399+0000 7fdcf602a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:44:43.465 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:43.471+0000 7fdcf602a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:44:43.691 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:43.697+0000 7fa2a8e28180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-27T06:44:43.762 INFO:tasks.ceph.mgr.x.smithi106.stderr:2024-05-27T06:44:43.769+0000 7fa2a8e28180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-27T06:44:44.488 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:44.495+0000 7fdc9b400640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-27T06:44:44.488 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T06:44:44.495+0000 7fdc9b400640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-27T06:44:45.674 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T06:44:45.681+0000 7fe646d437c0 -1 osd.4 0 log_to_monitors true 2024-05-27T06:44:45.823 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T06:44:45.829+0000 7fce083827c0 -1 osd.7 0 log_to_monitors true 2024-05-27T06:44:45.830 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T06:44:45.839+0000 7fdff035a7c0 -1 osd.3 0 log_to_monitors true 2024-05-27T06:44:45.943 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T06:44:45.949+0000 7f1337c397c0 -1 osd.6 0 log_to_monitors true 2024-05-27T06:44:45.956 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T06:44:45.963+0000 7fc08c02c7c0 -1 osd.0 0 log_to_monitors true 2024-05-27T06:44:46.057 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T06:44:46.065+0000 7f1af00127c0 -1 osd.5 0 log_to_monitors true 2024-05-27T06:44:46.071 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T06:44:46.079+0000 7efce0f387c0 -1 osd.1 0 log_to_monitors true 2024-05-27T06:44:46.147 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T06:44:46.155+0000 7f5d9fc937c0 -1 osd.2 0 log_to_monitors true 2024-05-27T06:44:49.089 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T06:44:49.097+0000 7f1aeae00640 -1 osd.5 0 waiting for initial osdmap 2024-05-27T06:44:49.093 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T06:44:49.099+0000 7efcdbe00640 -1 osd.1 0 waiting for initial osdmap 2024-05-27T06:44:49.112 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T06:44:49.119+0000 7fdfeb200640 -1 osd.3 0 waiting for initial osdmap 2024-05-27T06:44:49.115 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T06:44:49.121+0000 7fe642600640 -1 osd.4 0 waiting for initial osdmap 2024-05-27T06:44:49.118 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T06:44:49.127+0000 7fc087000640 -1 osd.0 0 waiting for initial osdmap 2024-05-27T06:44:49.119 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T06:44:49.125+0000 7fce03200640 -1 osd.7 0 waiting for initial osdmap 2024-05-27T06:44:49.124 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T06:44:49.131+0000 7efcd1e00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.126 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T06:44:49.133+0000 7f1ae0e00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.148 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T06:44:49.155+0000 7fdfe1200640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.154 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T06:44:49.161+0000 7fcdf9200640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.154 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T06:44:49.161+0000 7fe637c00640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.155 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T06:44:49.163+0000 7f5d9ac00640 -1 osd.2 0 waiting for initial osdmap 2024-05-27T06:44:49.155 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T06:44:49.161+0000 7f1332a00640 -1 osd.6 0 waiting for initial osdmap 2024-05-27T06:44:49.167 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T06:44:49.175+0000 7fc07c600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.189 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T06:44:49.197+0000 7f1328a00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.199 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T06:44:49.207+0000 7f5d90c00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-27T06:44:49.350 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T06:44:49.487 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 Processor -- start 2024-05-27T06:44:49.488 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 -- start start 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc80cae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fecc806e290 0x7fecc806e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecc80ca480 con 0x7fecc815d400 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fecc80ca720 con 0x7fecc806e290 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccfa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fecc80caa00 con 0x7fecc80c95d0 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccd000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fecc815d400 0x7fecc8160740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38562/0 (socket says 172.21.15.81:38562) 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccd000640 1 -- 172.21.15.81:0/2053082137 learned_addr learned my addr 172.21.15.81:0/2053082137 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7feccda00640 1 --2- 172.21.15.81:0/2053082137 >> v2:172.21.15.81:3300/0 conn(0x7fecc806e290 0x7fecc806e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.490 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7fecc7e00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc80cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7fecc7400640 1 -- 172.21.15.81:0/2053082137 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088393727 0 0) 0x7fecc80ca480 con 0x7fecc815d400 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.495+0000 7fecc7400640 1 -- 172.21.15.81:0/2053082137 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecb0003530 con 0x7fecc815d400 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccda00640 1 -- 172.21.15.81:0/2053082137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 msgr2=0x7fecc80cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccda00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc80cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccda00640 1 -- 172.21.15.81:0/2053082137 >> v1:172.21.15.81:6789/0 conn(0x7fecc815d400 legacy=0x7fecc8160740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.491 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccda00640 1 -- 172.21.15.81:0/2053082137 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc815c920 con 0x7fecc806e290 2024-05-27T06:44:49.492 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7fecc7e00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc80cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:49.493 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccda00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc806e6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009d90 tx=0x7fecc001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883d7a70d76f4d4f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:49.493 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7fecc7400640 1 -- 172.21.15.81:0/2053082137 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc00b4070 con 0x7fecc806e290 2024-05-27T06:44:49.493 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7fecc7400640 1 -- 172.21.15.81:0/2053082137 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc00af600 con 0x7fecc806e290 2024-05-27T06:44:49.494 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7fecc7400640 1 -- 172.21.15.81:0/2053082137 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc00af9d0 con 0x7fecc806e290 2024-05-27T06:44:49.494 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccfa00640 1 -- 172.21.15.81:0/2053082137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 msgr2=0x7fecc806e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.494 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.499+0000 7feccfa00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc806e6a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009d90 tx=0x7fecc001f150 comp rx=0 tx=0).stop 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- 172.21.15.81:0/2053082137 shutdown_connections 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc806e6a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 --2- 172.21.15.81:0/2053082137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc80cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- 172.21.15.81:0/2053082137 >> 172.21.15.81:0/2053082137 conn(0x7fecc80763e0 msgr2=0x7fecc8074480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- 172.21.15.81:0/2053082137 shutdown_connections 2024-05-27T06:44:49.495 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- 172.21.15.81:0/2053082137 wait complete. 2024-05-27T06:44:49.496 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 Processor -- start 2024-05-27T06:44:49.496 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- start start 2024-05-27T06:44:49.499 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.499 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc81e7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.499 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecc815d400 0x7fecc80c8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.499 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fecc80c9c20 con 0x7fecc806e290 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fecc815cee0 con 0x7fecc815d400 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.503+0000 7feccfa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fecc80c9da0 con 0x7fecc80c95d0 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53090/0 (socket says 172.21.15.81:53090) 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 -- 172.21.15.81:0/760072594 learned_addr learned my addr 172.21.15.81:0/760072594 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7fecc7e00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc81e7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccda00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecc815d400 0x7fecc80c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 -- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 msgr2=0x7fecc81e7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc81e7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 -- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecc815d400 msgr2=0x7fecc80c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecc815d400 0x7fecc80c8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 -- 172.21.15.81:0/760072594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc81e7790 con 0x7fecc806e290 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccd000640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00c430 tx=0x7fecbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc017070 con 0x7fecc806e290 2024-05-27T06:44:49.500 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecbc005cf0 con 0x7fecc806e290 2024-05-27T06:44:49.501 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc0072d0 con 0x7fecc806e290 2024-05-27T06:44:49.501 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc815ea10 con 0x7fecc806e290 2024-05-27T06:44:49.501 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc815efd0 con 0x7fecc806e290 2024-05-27T06:44:49.501 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc80774a0 con 0x7fecc806e290 2024-05-27T06:44:49.501 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.507+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) ==== 93151+0+0 (secure 0 0 0) 0x7fecbc01a070 con 0x7fecc806e290 2024-05-27T06:44:49.502 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.511+0000 7fecc4c00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 0x7feca8071880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:49.502 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.511+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fecbc08d650 con 0x7fecc806e290 2024-05-27T06:44:49.503 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.511+0000 7fecc7e00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 0x7feca8071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:49.503 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.511+0000 7fecc7e00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 0x7feca8071880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fecc81f3a90 tx=0x7fecb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:49.508 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.515+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fecbc05ae60 con 0x7fecc806e290 2024-05-27T06:44:49.670 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecc806de60 con 0x7fecc806e290 2024-05-27T06:44:49.671 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7fecc4c00640 1 -- 172.21.15.81:0/760072594 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fecbc05f160 con 0x7fecc806e290 2024-05-27T06:44:49.671 INFO:teuthology.misc.health.smithi081.stdout: 2024-05-27T06:44:49.671 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":12,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:44:47.596658+0000","last_up_change":"0.000000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:44:49.673 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 msgr2=0x7feca8071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.673 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 0x7feca8071880 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fecc81f3a90 tx=0x7fecb4008040 comp rx=0 tx=0).stop 2024-05-27T06:44:49.673 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 msgr2=0x7fecc815f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:49.673 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00c430 tx=0x7fecbc00c8f0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 shutdown_connections 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fecc80c95d0 0x7fecc81e7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feca806f370 0x7feca8071880 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecc815d400 0x7fecc80c8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 --2- 172.21.15.81:0/760072594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecc806e290 0x7fecc815f8d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 >> 172.21.15.81:0/760072594 conn(0x7fecc80763e0 msgr2=0x7fecc8151eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 shutdown_connections 2024-05-27T06:44:49.674 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:49.679+0000 7feccfa00640 1 -- 172.21.15.81:0/760072594 wait complete. 2024-05-27T06:44:49.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-27T06:44:56.496 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-27T06:44:56.646 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.651+0000 7f81e4800640 1 Processor -- start 2024-05-27T06:44:56.646 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.651+0000 7f81e4800640 1 -- start start 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e4800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f81dc0c9140 0x7f81dc160440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e4800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e4800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81dc15cc20 con 0x7f81dc0c9550 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e4800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f81dc15cf00 con 0x7f81dc0c9140 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e4800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f81dc15d1e0 con 0x7f81dc160980 2024-05-27T06:44:56.648 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f81dc0c9140 0x7f81dc160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f81dc0c9140 0x7f81dc160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47688/0 (socket says 172.21.15.81:47688) 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 -- 172.21.15.81:0/2964758376 learned_addr learned my addr 172.21.15.81:0/2964758376 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1400640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 -- 172.21.15.81:0/2964758376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 msgr2=0x7f81dc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 -- 172.21.15.81:0/2964758376 >> v1:172.21.15.106:6789/0 conn(0x7f81dc0c9550 legacy=0x7f81dc070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.649 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.655+0000 7f81e1e00640 1 -- 172.21.15.81:0/2964758376 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81dc15d4c0 con 0x7f81dc0c9140 2024-05-27T06:44:56.651 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e1e00640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9140 0x7f81dc160440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81d0006fd0 tx=0x7f81d001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a9549f27412417 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:56.652 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e0a00640 1 -- 172.21.15.81:0/2964758376 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00b5070 con 0x7f81dc0c9140 2024-05-27T06:44:56.652 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e0a00640 1 -- 172.21.15.81:0/2964758376 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d00a8a30 con 0x7f81dc0c9140 2024-05-27T06:44:56.652 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e0a00640 1 -- 172.21.15.81:0/2964758376 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00a8e00 con 0x7f81dc0c9140 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 -- 172.21.15.81:0/2964758376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9140 msgr2=0x7f81dc160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9140 0x7f81dc160440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f81d0006fd0 tx=0x7f81d001ed30 comp rx=0 tx=0).stop 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 -- 172.21.15.81:0/2964758376 shutdown_connections 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 --2- 172.21.15.81:0/2964758376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9140 0x7f81dc160440 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 -- 172.21.15.81:0/2964758376 >> 172.21.15.81:0/2964758376 conn(0x7f81dc06dea0 msgr2=0x7f81dc0ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 -- 172.21.15.81:0/2964758376 shutdown_connections 2024-05-27T06:44:56.653 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.659+0000 7f81e4800640 1 -- 172.21.15.81:0/2964758376 wait complete. 2024-05-27T06:44:56.655 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 Processor -- start 2024-05-27T06:44:56.655 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- start start 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81dc0c9140 0x7f81dc1576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc073360 con 0x7f81dc0c9550 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc0731e0 con 0x7f81dc0c9140 2024-05-27T06:44:56.657 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f81dc156ae0 con 0x7f81dc160980 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47698/0 (socket says 172.21.15.81:47698) 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 -- 172.21.15.81:0/1255420450 learned_addr learned my addr 172.21.15.81:0/1255420450 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1e00640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81dc0c9140 0x7f81dc1576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e2800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 -- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 msgr2=0x7f81dc14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 -- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81dc0c9140 msgr2=0x7f81dc1576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81dc0c9140 0x7f81dc1576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.658 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 -- 172.21.15.81:0/1255420450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81dc157c10 con 0x7f81dc0c9550 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e1400640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e840 tx=0x7f81cc00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc017070 con 0x7f81dc0c9550 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc00ee90 con 0x7f81dc0c9550 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81dc159b30 con 0x7f81dc0c9550 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00cb80 con 0x7f81dc0c9550 2024-05-27T06:44:56.659 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.663+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81dc159dd0 con 0x7f81dc0c9550 2024-05-27T06:44:56.660 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81dc0774a0 con 0x7f81dc0c9550 2024-05-27T06:44:56.660 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f81cc01a070 con 0x7f81dc0c9550 2024-05-27T06:44:56.661 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81d6000640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 0x7f81c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:56.661 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f81cc08ede0 con 0x7f81dc0c9550 2024-05-27T06:44:56.661 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81e1e00640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 0x7f81c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:56.662 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.667+0000 7f81e1e00640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 0x7f81c0071910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f81d00b2000 tx=0x7f81d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:56.666 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.675+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f81cc05b9c0 con 0x7f81dc0c9550 2024-05-27T06:44:56.826 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.831+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81dc1f3d30 con 0x7f81dc0c9550 2024-05-27T06:44:56.827 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81d6000640 1 -- 172.21.15.81:0/1255420450 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f81cc05fcc0 con 0x7f81dc0c9550 2024-05-27T06:44:56.827 INFO:teuthology.misc.health.smithi081.stdout: 2024-05-27T06:44:56.827 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":17,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:44:54.159004+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:44:56.829 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 msgr2=0x7f81c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.829 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 0x7f81c0071910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f81d00b2000 tx=0x7f81d0009040 comp rx=0 tx=0).stop 2024-05-27T06:44:56.829 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 msgr2=0x7f81dc0ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:56.829 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e840 tx=0x7f81cc00ec40 comp rx=0 tx=0).stop 2024-05-27T06:44:56.829 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 shutdown_connections 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f81dc160980 0x7f81dc14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f81c006f400 0x7f81c0071910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81dc0c9550 0x7f81dc0ca8b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 --2- 172.21.15.81:0/1255420450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f81dc0c9140 0x7f81dc1576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 >> 172.21.15.81:0/1255420450 conn(0x7f81dc06dea0 msgr2=0x7f81dc15b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 shutdown_connections 2024-05-27T06:44:56.830 INFO:teuthology.misc.health.smithi081.stderr:2024-05-27T06:44:56.835+0000 7f81e4800640 1 -- 172.21.15.81:0/1255420450 wait complete. 2024-05-27T06:44:56.847 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-27T06:44:56.847 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-27T06:44:56.847 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T06:44:56.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T06:44:56.850 INFO:tasks.ceph.mds.a:Started 2024-05-27T06:44:56.850 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T06:44:56.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T06:44:56.853 INFO:tasks.ceph.mds.c:Started 2024-05-27T06:44:56.853 INFO:tasks.ceph.mds.e:Restarting daemon 2024-05-27T06:44:56.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-05-27T06:44:56.856 INFO:tasks.ceph.mds.e:Started 2024-05-27T06:44:56.856 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-27T06:44:56.857 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-27T06:44:56.860 INFO:tasks.ceph.mds.b:Started 2024-05-27T06:44:56.860 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-27T06:44:56.860 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-27T06:44:56.863 INFO:tasks.ceph.mds.d:Started 2024-05-27T06:44:56.863 INFO:tasks.ceph.mds.f:Restarting daemon 2024-05-27T06:44:56.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-05-27T06:44:56.867 INFO:tasks.ceph.mds.f:Started 2024-05-27T06:44:56.867 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-27T06:44:56.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:44:56.966 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-05-27T06:44:56.967 INFO:tasks.ceph.mds.e.smithi081.stdout:starting mds.e at 2024-05-27T06:44:56.969 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-05-27T06:44:56.984 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-05-27T06:44:56.985 INFO:tasks.ceph.mds.f.smithi106.stdout:starting mds.f at 2024-05-27T06:44:56.986 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-05-27T06:44:57.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.039+0000 7f8d99c00640 1 Processor -- start 2024-05-27T06:44:57.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.039+0000 7f8d99c00640 1 -- start start 2024-05-27T06:44:57.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d99c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d940c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d99c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8d94161cd0 0x7f8d941640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d99c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d94072840 con 0x7f8d94072430 2024-05-27T06:44:57.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d99c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8d940c5040 con 0x7f8d94161cd0 2024-05-27T06:44:57.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d99c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8d940c75c0 con 0x7f8d940c5b60 2024-05-27T06:44:57.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8d94161cd0 0x7f8d941640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8d94161cd0 0x7f8d941640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47734/0 (socket says 172.21.15.81:47734) 2024-05-27T06:44:57.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 -- 172.21.15.81:0/1376558966 learned_addr learned my addr 172.21.15.81:0/1376558966 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d8a000640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d940c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 -- 172.21.15.81:0/1376558966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 msgr2=0x7f8d940c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d940c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 -- 172.21.15.81:0/1376558966 >> v1:172.21.15.106:6789/0 conn(0x7f8d94072430 legacy=0x7f8d94159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d93400640 1 -- 172.21.15.81:0/1376558966 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d940c5270 con 0x7f8d94161cd0 2024-05-27T06:44:57.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.043+0000 7f8d8a000640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d940c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:44:57.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d93400640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d941640f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009e50 tx=0x7f8d8c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d37575503d2595 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:57.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d92000640 1 -- 172.21.15.81:0/1376558966 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0a5670 con 0x7f8d94161cd0 2024-05-27T06:44:57.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d92000640 1 -- 172.21.15.81:0/1376558966 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8c0a5810 con 0x7f8d94161cd0 2024-05-27T06:44:57.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d92000640 1 -- 172.21.15.81:0/1376558966 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0a5be0 con 0x7f8d94161cd0 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- 172.21.15.81:0/1376558966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 msgr2=0x7f8d941640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d941640f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009e50 tx=0x7f8d8c01ca30 comp rx=0 tx=0).stop 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- 172.21.15.81:0/1376558966 shutdown_connections 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d941640f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1376558966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d940c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- 172.21.15.81:0/1376558966 >> 172.21.15.81:0/1376558966 conn(0x7f8d94070700 msgr2=0x7f8d940cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- 172.21.15.81:0/1376558966 shutdown_connections 2024-05-27T06:44:57.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- 172.21.15.81:0/1376558966 wait complete. 2024-05-27T06:44:57.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 Processor -- start 2024-05-27T06:44:57.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.047+0000 7f8d99c00640 1 -- start start 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 0x7f8d94150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d9414eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8d940ce6f0 con 0x7f8d94161cd0 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8d940c7dd0 con 0x7f8d94072430 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8d940c5610 con 0x7f8d940c5b60 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47736/0 (socket says 172.21.15.81:47736) 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 -- 172.21.15.81:0/1531656745 learned_addr learned my addr 172.21.15.81:0/1531656745 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d92a00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 0x7f8d94150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d8a000640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d9414eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 -- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 msgr2=0x7f8d9414eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:57.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d9414eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 -- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 msgr2=0x7f8d94150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 0x7f8d94150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 -- 172.21.15.81:0/1531656745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d941522e0 con 0x7f8d94161cd0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d92a00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 0x7f8d94150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d93400640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0af040 tx=0x7f8d8c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c009510 con 0x7f8d94161cd0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8c0be070 con 0x7f8d94161cd0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9414f410 con 0x7f8d94161cd0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c0b88d0 con 0x7f8d94161cd0 2024-05-27T06:44:57.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d941577f0 con 0x7f8d94161cd0 2024-05-27T06:44:57.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.051+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9406a560 con 0x7f8d94161cd0 2024-05-27T06:44:57.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.055+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8d8c0b8a70 con 0x7f8d94161cd0 2024-05-27T06:44:57.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.055+0000 7f8d8b400640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 0x7f8d70071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:57.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.055+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f8d8c12a030 con 0x7f8d94161cd0 2024-05-27T06:44:57.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.055+0000 7f8d92a00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 0x7f8d70071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:57.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.055+0000 7f8d92a00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 0x7f8d70071910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d94070fe0 tx=0x7f8d84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:57.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.059+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8d941522e0 con 0x7f8d94161cd0 2024-05-27T06:44:57.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.251+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d94152bf0 con 0x7f8d94161cd0 2024-05-27T06:44:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.251+0000 7f8d8b400640 1 -- 172.21.15.81:0/1531656745 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f8d8c016620 con 0x7f8d94161cd0 2024-05-27T06:44:57.246 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:57.246 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-27T06:44:57.246 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 2 2024-05-27T06:44:57.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 msgr2=0x7f8d70071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:57.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 0x7f8d70071910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8d94070fe0 tx=0x7f8d84008040 comp rx=0 tx=0).stop 2024-05-27T06:44:57.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 msgr2=0x7f8d9414b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:57.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0af040 tx=0x7f8d8c0197d0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 shutdown_connections 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d940c5b60 0x7f8d9414eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7006f400 0x7f8d70071910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d94161cd0 0x7f8d9414b5d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 --2- 172.21.15.81:0/1531656745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d94072430 0x7f8d94150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 >> 172.21.15.81:0/1531656745 conn(0x7f8d94070700 msgr2=0x7f8d94142e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 shutdown_connections 2024-05-27T06:44:57.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:57.255+0000 7f8d99c00640 1 -- 172.21.15.81:0/1531656745 wait complete. 2024-05-27T06:44:59.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:44:59.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 Processor -- start 2024-05-27T06:44:59.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 -- start start 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 0x7fbc20152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc2006c790 0x7fbc2006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc200c9f80 con 0x7fbc200c8880 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fbc200c58b0 con 0x7fbc2006c790 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc25e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200c83f0 con 0x7fbc2015cb80 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc2006c790 0x7fbc2006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc2006c790 0x7fbc2006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47750/0 (socket says 172.21.15.81:47750) 2024-05-27T06:44:59.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 -- 172.21.15.81:0/2060538078 learned_addr learned my addr 172.21.15.81:0/2060538078 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc1ea00640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 0x7fbc20152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 -- 172.21.15.81:0/2060538078 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 msgr2=0x7fbc20152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 0x7fbc20152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 -- 172.21.15.81:0/2060538078 >> v1:172.21.15.81:6789/0 conn(0x7fbc200c8880 legacy=0x7fbc200c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc16000640 1 -- 172.21.15.81:0/2060538078 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc200c52f0 con 0x7fbc2006c790 2024-05-27T06:44:59.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.434+0000 7fbc1ea00640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 0x7fbc20152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:44:59.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc16000640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2006c790 0x7fbc2006cba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009e50 tx=0x7fbc0801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fea81901a20b79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc1e000640 1 -- 172.21.15.81:0/2060538078 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc080a5700 con 0x7fbc2006c790 2024-05-27T06:44:59.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc1e000640 1 -- 172.21.15.81:0/2060538078 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc080a58a0 con 0x7fbc2006c790 2024-05-27T06:44:59.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc1e000640 1 -- 172.21.15.81:0/2060538078 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc080a5c70 con 0x7fbc2006c790 2024-05-27T06:44:59.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 -- 172.21.15.81:0/2060538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2006c790 msgr2=0x7fbc2006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2006c790 0x7fbc2006cba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009e50 tx=0x7fbc0801ca30 comp rx=0 tx=0).stop 2024-05-27T06:44:59.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 -- 172.21.15.81:0/2060538078 shutdown_connections 2024-05-27T06:44:59.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2006c790 0x7fbc2006cba0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 --2- 172.21.15.81:0/2060538078 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2015cb80 0x7fbc20152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 -- 172.21.15.81:0/2060538078 >> 172.21.15.81:0/2060538078 conn(0x7fbc20070700 msgr2=0x7fbc200cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:59.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 -- 172.21.15.81:0/2060538078 shutdown_connections 2024-05-27T06:44:59.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.438+0000 7fbc25e00640 1 -- 172.21.15.81:0/2060538078 wait complete. 2024-05-27T06:44:59.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 Processor -- start 2024-05-27T06:44:59.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- start start 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2006c790 0x7fbc20144a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 0x7fbc200cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200c9c80 con 0x7fbc2015cb80 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200c9b00 con 0x7fbc200c8880 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fbc20165d00 con 0x7fbc2006c790 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47766/0 (socket says 172.21.15.81:47766) 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 -- 172.21.15.81:0/3744889366 learned_addr learned my addr 172.21.15.81:0/3744889366 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:59.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1ea00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2006c790 0x7fbc20144a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc16000640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 0x7fbc200cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 -- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2006c790 msgr2=0x7fbc20144a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2006c790 0x7fbc20144a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 -- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 msgr2=0x7fbc200cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 0x7fbc200cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 -- 172.21.15.81:0/3744889366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc20154500 con 0x7fbc2015cb80 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc16000640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 0x7fbc200cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:44:59.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc1f400640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800c3f0 tx=0x7fbc1800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18017070 con 0x7fbc2015cb80 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc180052f0 con 0x7fbc2015cb80 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc20152540 con 0x7fbc2015cb80 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18003860 con 0x7fbc2015cb80 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.442+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc20153a20 con 0x7fbc2015cb80 2024-05-27T06:44:59.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2006a560 con 0x7fbc2015cb80 2024-05-27T06:44:59.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbc1801a070 con 0x7fbc2015cb80 2024-05-27T06:44:59.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc17e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 0x7fbbfc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fbc1808e270 con 0x7fbc2015cb80 2024-05-27T06:44:59.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc1ea00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 0x7fbbfc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.446+0000 7fbc1ea00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 0x7fbbfc071910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc10006fd0 tx=0x7fbc10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.450+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fbc1805adc0 con 0x7fbc2015cb80 2024-05-27T06:44:59.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.642+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc201549e0 con 0x7fbc2015cb80 2024-05-27T06:44:59.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc17e00640 1 -- 172.21.15.81:0/3744889366 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5506 (secure 0 0 0) 0x7fbc1805f0c0 con 0x7fbc2015cb80 2024-05-27T06:44:59.639 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:59.639 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4302,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-27T06:44:59.640 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 3 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 msgr2=0x7fbbfc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 0x7fbbfc071910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc10006fd0 tx=0x7fbc10008040 comp rx=0 tx=0).stop 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 msgr2=0x7fbc201402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800c3f0 tx=0x7fbc1800c7f0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 shutdown_connections 2024-05-27T06:44:59.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbbfc06f400 0x7fbbfc071910 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc2015cb80 0x7fbc201402c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc200c8880 0x7fbc200cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 --2- 172.21.15.81:0/3744889366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc2006c790 0x7fbc20144a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 >> 172.21.15.81:0/3744889366 conn(0x7fbc20070700 msgr2=0x7fbc200cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 shutdown_connections 2024-05-27T06:44:59.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.646+0000 7fbc25e00640 1 -- 172.21.15.81:0/3744889366 wait complete. 2024-05-27T06:44:59.661 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-27T06:44:59.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 Processor -- start 2024-05-27T06:44:59.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 -- start start 2024-05-27T06:44:59.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f51e0166160 0x7f51e0168580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e00ca1e0 con 0x7f51e00c9710 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e00d0860 con 0x7f51e015fe80 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51e5400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f51e015d5a0 con 0x7f51e0166160 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.814+0000 7f51df200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f51e0166160 0x7f51e0168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f51e0166160 0x7f51e0168580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47776/0 (socket says 172.21.15.81:47776) 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 -- 172.21.15.81:0/1829838283 learned_addr learned my addr 172.21.15.81:0/1829838283 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 -- 172.21.15.81:0/1829838283 >> v1:172.21.15.106:6789/0 conn(0x7f51e015fe80 legacy=0x7f51e0160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 -- 172.21.15.81:0/1829838283 >> v1:172.21.15.81:6789/0 conn(0x7f51e00c9710 legacy=0x7f51e0072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 -- 172.21.15.81:0/1829838283 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e00ca4a0 con 0x7f51e0166160 2024-05-27T06:44:59.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51dde00640 1 -- 172.21.15.81:0/1829838283 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1746411140 0 0) 0x7f51e00ca1e0 con 0x7f51e00c9710 2024-05-27T06:44:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51df200640 1 --2- 172.21.15.81:0/1829838283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e0166160 0x7f51e0168580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009e50 tx=0x7f51d801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f4c0243714ade server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51dde00640 1 -- 172.21.15.81:0/1829838283 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d80a5900 con 0x7f51e0166160 2024-05-27T06:44:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51dde00640 1 -- 172.21.15.81:0/1829838283 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d80a5aa0 con 0x7f51e0166160 2024-05-27T06:44:59.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.818+0000 7f51dde00640 1 -- 172.21.15.81:0/1829838283 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d80ad3f0 con 0x7f51e0166160 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- 172.21.15.81:0/1829838283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e0166160 msgr2=0x7f51e0168580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 --2- 172.21.15.81:0/1829838283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e0166160 0x7f51e0168580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009e50 tx=0x7f51d801ca30 comp rx=0 tx=0).stop 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- 172.21.15.81:0/1829838283 shutdown_connections 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 --2- 172.21.15.81:0/1829838283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e0166160 0x7f51e0168580 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- 172.21.15.81:0/1829838283 >> 172.21.15.81:0/1829838283 conn(0x7f51e0068cb0 msgr2=0x7f51e00cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- 172.21.15.81:0/1829838283 shutdown_connections 2024-05-27T06:44:59.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- 172.21.15.81:0/1829838283 wait complete. 2024-05-27T06:44:59.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 Processor -- start 2024-05-27T06:44:59.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.822+0000 7f51e5400640 1 -- start start 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 0x7f51e00cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 0x7f51e0167dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e0166160 0x7f51e01f9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51e00ca7b0 con 0x7f51e015fe80 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f51e015d230 con 0x7f51e0166160 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f51e00ca930 con 0x7f51e00c9710 2024-05-27T06:44:59.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51de800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 0x7f51e00cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51df200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e0166160 0x7f51e01f9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51de800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 0x7f51e00cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:52388/0 (socket says 172.21.15.81:52388) 2024-05-27T06:44:59.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51de800640 1 -- 172.21.15.81:0/2269764728 learned_addr learned my addr 172.21.15.81:0/2269764728 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:44:59.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 0x7f51e0167dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 -- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 msgr2=0x7f51e00cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 0x7f51e00cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 -- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e0166160 msgr2=0x7f51e01f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e0166160 0x7f51e01f9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 -- 172.21.15.81:0/2269764728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e015bc60 con 0x7f51e015fe80 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d5e00640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 0x7f51e0167dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c3f0 tx=0x7f51c800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8017070 con 0x7f51e015fe80 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c80052d0 con 0x7f51e015fe80 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8003820 con 0x7f51e015fe80 2024-05-27T06:44:59.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e015c6e0 con 0x7f51e015fe80 2024-05-27T06:44:59.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e015bee0 con 0x7f51e015fe80 2024-05-27T06:44:59.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.826+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e006a560 con 0x7f51e015fe80 2024-05-27T06:44:59.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.830+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51c801a070 con 0x7f51e015fe80 2024-05-27T06:44:59.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.830+0000 7f51d7400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 0x7f51bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:44:59.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.830+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f51c808e280 con 0x7f51e015fe80 2024-05-27T06:44:59.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.830+0000 7f51de800640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 0x7f51bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:44:59.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.830+0000 7f51de800640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 0x7f51bc071910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d00017f0 tx=0x7f51d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:44:59.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.834+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f51e015bc60 con 0x7f51e015fe80 2024-05-27T06:44:59.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f51e00cab30 con 0x7f51e015fe80 2024-05-27T06:44:59.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51d7400640 1 -- 172.21.15.81:0/2269764728 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f51c8005480 con 0x7f51e015fe80 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-05-27T06:44:59.989 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 msgr2=0x7f51bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 0x7f51bc071910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d00017f0 tx=0x7f51d0009040 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 msgr2=0x7f51e0167dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 0x7f51e0167dc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f51c800c3f0 tx=0x7f51c800c8b0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 shutdown_connections 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51bc06f400 0x7f51bc071910 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e0166160 0x7f51e01f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e015fe80 0x7f51e0167dc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 --2- 172.21.15.81:0/2269764728 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e00c9710 0x7f51e00cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.994+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 >> 172.21.15.81:0/2269764728 conn(0x7f51e0068cb0 msgr2=0x7f51e00ce530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.998+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 shutdown_connections 2024-05-27T06:44:59.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:44:59.998+0000 7f51e5400640 1 -- 172.21.15.81:0/2269764728 wait complete. 2024-05-27T06:45:00.010 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-27T06:45:00.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-27T06:45:00.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.166+0000 7fd46ac00640 1 Processor -- start 2024-05-27T06:45:00.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.166+0000 7fd46ac00640 1 -- start start 2024-05-27T06:45:00.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd46ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd464074ac0 0x7fd464074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd46ac00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 0x7fd46406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd46ac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd46406dff0 con 0x7fd46415fe30 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd46ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd4640cac00 con 0x7fd464074ac0 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd46ac00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd464160600 con 0x7fd46406ea60 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd464074ac0 0x7fd464074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd46415fe30 0x7fd4640ca4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38548/0 (socket says 172.21.15.81:38548) 2024-05-27T06:45:00.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463e00640 1 -- 172.21.15.81:0/3658040816 learned_addr learned my addr 172.21.15.81:0/3658040816 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:00.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd468c00640 1 --2- 172.21.15.81:0/3658040816 >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 0x7fd46406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463400640 1 -- 172.21.15.81:0/3658040816 >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 msgr2=0x7fd46406ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:00.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463400640 1 --2- 172.21.15.81:0/3658040816 >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 0x7fd46406ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463400640 1 -- 172.21.15.81:0/3658040816 >> v1:172.21.15.81:6789/0 conn(0x7fd46415fe30 legacy=0x7fd4640ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd463400640 1 -- 172.21.15.81:0/3658040816 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4641608e0 con 0x7fd464074ac0 2024-05-27T06:45:00.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd462a00640 1 -- 172.21.15.81:0/3658040816 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2290557036 0 0) 0x7fd46406dff0 con 0x7fd46415fe30 2024-05-27T06:45:00.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.170+0000 7fd468c00640 1 --2- 172.21.15.81:0/3658040816 >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 0x7fd46406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:00.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd463400640 1 --2- 172.21.15.81:0/3658040816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd464074ac0 0x7fd464074ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd46406fcd0 tx=0x7fd45001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46476a715c1de659 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd462a00640 1 -- 172.21.15.81:0/3658040816 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4500b0070 con 0x7fd464074ac0 2024-05-27T06:45:00.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd462a00640 1 -- 172.21.15.81:0/3658040816 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4500ab600 con 0x7fd464074ac0 2024-05-27T06:45:00.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd462a00640 1 -- 172.21.15.81:0/3658040816 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4500ab9d0 con 0x7fd464074ac0 2024-05-27T06:45:00.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 -- 172.21.15.81:0/3658040816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd464074ac0 msgr2=0x7fd464074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3658040816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd464074ac0 0x7fd464074ed0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd46406fcd0 tx=0x7fd45001c480 comp rx=0 tx=0).stop 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 -- 172.21.15.81:0/3658040816 shutdown_connections 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3658040816 >> v2:172.21.15.106:3300/0 conn(0x7fd46406ea60 0x7fd46406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3658040816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd464074ac0 0x7fd464074ed0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 -- 172.21.15.81:0/3658040816 >> 172.21.15.81:0/3658040816 conn(0x7fd464072500 msgr2=0x7fd464072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 -- 172.21.15.81:0/3658040816 shutdown_connections 2024-05-27T06:45:00.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.174+0000 7fd46ac00640 1 -- 172.21.15.81:0/3658040816 wait complete. 2024-05-27T06:45:00.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.178+0000 7fd46ac00640 1 Processor -- start 2024-05-27T06:45:00.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.178+0000 7fd46ac00640 1 -- start start 2024-05-27T06:45:00.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.178+0000 7fd46ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd464074ac0 0x7fd46415ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd46415fe30 0x7fd46414d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd464160240 con 0x7fd46406ea60 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd46406f3b0 con 0x7fd464074ac0 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd4641603c0 con 0x7fd46415fe30 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47786/0 (socket says 172.21.15.81:47786) 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 -- 172.21.15.81:0/3054872597 learned_addr learned my addr 172.21.15.81:0/3054872597 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd468c00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd46415fe30 0x7fd46414d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463400640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd464074ac0 0x7fd46415ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 -- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd46415fe30 msgr2=0x7fd46414d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd46415fe30 0x7fd46414d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 -- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd464074ac0 msgr2=0x7fd46415ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd464074ac0 0x7fd46415ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 -- 172.21.15.81:0/3054872597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4641f9cd0 con 0x7fd46406ea60 2024-05-27T06:45:00.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd463e00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd454005890 tx=0x7fd45400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454017070 con 0x7fd46406ea60 2024-05-27T06:45:00.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454005d10 con 0x7fd46406ea60 2024-05-27T06:45:00.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454007310 con 0x7fd46406ea60 2024-05-27T06:45:00.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4641f5f80 con 0x7fd46406ea60 2024-05-27T06:45:00.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4641f99f0 con 0x7fd46406ea60 2024-05-27T06:45:00.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.182+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46406a560 con 0x7fd46406ea60 2024-05-27T06:45:00.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.186+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd45401a070 con 0x7fd46406ea60 2024-05-27T06:45:00.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.186+0000 7fd45be00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 0x7fd444071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.186+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd45408e5b0 con 0x7fd46406ea60 2024-05-27T06:45:00.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.186+0000 7fd463400640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 0x7fd444071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.186+0000 7fd463400640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 0x7fd444071910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4500ad000 tx=0x7fd450008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.190+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd454021020 con 0x7fd46406ea60 2024-05-27T06:45:00.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.346+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd4640c52a0 con 0x7fd46406ea60 2024-05-27T06:45:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.590+0000 7fd45be00640 1 -- 172.21.15.81:0/3054872597 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fd45405b310 con 0x7fd46406ea60 2024-05-27T06:45:00.587 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 msgr2=0x7fd444071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 0x7fd444071910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd4500ad000 tx=0x7fd450008040 comp rx=0 tx=0).stop 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 msgr2=0x7fd4641f5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd454005890 tx=0x7fd45400c890 comp rx=0 tx=0).stop 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 shutdown_connections 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd46415fe30 0x7fd46414d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd44406f400 0x7fd444071910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd464074ac0 0x7fd46415ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 --2- 172.21.15.81:0/3054872597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46406ea60 0x7fd4641f5a40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 >> 172.21.15.81:0/3054872597 conn(0x7fd464072500 msgr2=0x7fd464151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:00.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 shutdown_connections 2024-05-27T06:45:00.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.594+0000 7fd46ac00640 1 -- 172.21.15.81:0/3054872597 wait complete. 2024-05-27T06:45:00.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-27T06:45:00.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 Processor -- start 2024-05-27T06:45:00.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 -- start start 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc0b40c73e0 0x7fc0b4165f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b4073ac0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b40ce080 con 0x7fc0b4154910 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b8e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc0b4154d20 con 0x7fc0b40c73e0 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0a9400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc0b40c6fd0 0x7fc0b415db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38564/0 (socket says 172.21.15.81:38564) 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0a9400640 1 -- 172.21.15.81:0/2368977725 learned_addr learned my addr 172.21.15.81:0/2368977725 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b2800640 1 --2- 172.21.15.81:0/2368977725 >> v2:172.21.15.106:3300/0 conn(0x7fc0b40c73e0 0x7fc0b4165f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3697144949 0 0) 0x7fc0b4073ac0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc098003530 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3106833076 0 0) 0x7fc098003530 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b2800640 1 -- 172.21.15.81:0/2368977725 >> v1:172.21.15.106:6789/0 conn(0x7fc0b4154910 legacy=0x7fc0b4074ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b2800640 1 -- 172.21.15.81:0/2368977725 >> v1:172.21.15.81:6789/0 conn(0x7fc0b40c6fd0 legacy=0x7fc0b415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.778+0000 7fc0b2800640 1 -- 172.21.15.81:0/2368977725 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b40c8780 con 0x7fc0b40c73e0 2024-05-27T06:45:00.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b2800640 1 --2- 172.21.15.81:0/2368977725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b4165f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4154ea0 tx=0x7fc0ac01c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4ea17d915e2bd15 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c001700 con 0x7fc0b40c73e0 2024-05-27T06:45:00.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0ac0a6430 con 0x7fc0b40c73e0 2024-05-27T06:45:00.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b1400640 1 -- 172.21.15.81:0/2368977725 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0a67e0 con 0x7fc0b40c73e0 2024-05-27T06:45:00.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 -- 172.21.15.81:0/2368977725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 msgr2=0x7fc0b4165f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/2368977725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b4165f70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4154ea0 tx=0x7fc0ac01c7e0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 -- 172.21.15.81:0/2368977725 shutdown_connections 2024-05-27T06:45:00.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/2368977725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b4165f70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 -- 172.21.15.81:0/2368977725 >> 172.21.15.81:0/2368977725 conn(0x7fc0b4070760 msgr2=0x7fc0b40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:00.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 -- 172.21.15.81:0/2368977725 shutdown_connections 2024-05-27T06:45:00.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.782+0000 7fc0b8e00640 1 -- 172.21.15.81:0/2368977725 wait complete. 2024-05-27T06:45:00.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 Processor -- start 2024-05-27T06:45:00.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 -- start start 2024-05-27T06:45:00.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b41635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 0x7fc0b4160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b40c77f0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b40735a0 con 0x7fc0b40c73e0 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b8e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b415e290 con 0x7fc0b4154910 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47792/0 (socket says 172.21.15.81:47792) 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b1e00640 1 -- 172.21.15.81:0/1369772205 learned_addr learned my addr 172.21.15.81:0/1369772205 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0a9400640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b41635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.786+0000 7fc0b2800640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 0x7fc0b4160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 -- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 msgr2=0x7fc0b4160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 0x7fc0b4160b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 -- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 msgr2=0x7fc0b41635d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b41635d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 -- 172.21.15.81:0/1369772205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b4161050 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0a9400640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b41635d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:00.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b2800640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 0x7fc0b4160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b1e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400c450 tx=0x7fc0a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40052d0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a4005470 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b41f5dc0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4012850 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b41f6380 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0b406a560 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.790+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc0a40129f0 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.794+0000 7fc0aaa00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 0x7fc090071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:00.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.794+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fc0a408f740 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.794+0000 7fc0a9400640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 0x7fc090071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:00.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.794+0000 7fc0a9400640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 0x7fc090071940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4159810 tx=0x7fc09c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:00.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.798+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc0a405c880 con 0x7fc0b40c6fd0 2024-05-27T06:45:00.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:00.954+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc0b4072a10 con 0x7fc0b40c6fd0 2024-05-27T06:45:01.398 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:45:01.402 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:45:01.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.602+0000 7fc0aaa00640 1 -- 172.21.15.81:0/1369772205 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fc0a405c180 con 0x7fc0b40c6fd0 2024-05-27T06:45:01.597 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 msgr2=0x7fc090071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 0x7fc090071940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4159810 tx=0x7fc09c009040 comp rx=0 tx=0).stop 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 msgr2=0x7fc0b41650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400c450 tx=0x7fc0a400c910 comp rx=0 tx=0).stop 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 shutdown_connections 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0b4154910 0x7fc0b4160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc09006f430 0x7fc090071940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0b40c73e0 0x7fc0b41635d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 --2- 172.21.15.81:0/1369772205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b40c6fd0 0x7fc0b41650e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 >> 172.21.15.81:0/1369772205 conn(0x7fc0b4070760 msgr2=0x7fc0b414f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 shutdown_connections 2024-05-27T06:45:01.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.606+0000 7fc0b8e00640 1 -- 172.21.15.81:0/1369772205 wait complete. 2024-05-27T06:45:01.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-27T06:45:01.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 Processor -- start 2024-05-27T06:45:01.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 -- start start 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f21ec1549f0 0x7f21ec15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ec06eac0 con 0x7f21ec15b1c0 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ec0ca9c0 con 0x7f21ec1545e0 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f3c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f21ec15db40 con 0x7f21ec1549f0 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f1c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f21ec15b1c0 0x7f21ec0c8290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38576/0 (socket says 172.21.15.81:38576) 2024-05-27T06:45:01.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21f1c00640 1 -- 172.21.15.81:0/2057156003 learned_addr learned my addr 172.21.15.81:0/2057156003 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:01.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21e3e00640 1 --2- 172.21.15.81:0/2057156003 >> v2:172.21.15.106:3300/0 conn(0x7f21ec1549f0 0x7f21ec15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:01.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737170828 0 0) 0x7f21ec06eac0 con 0x7f21ec15b1c0 2024-05-27T06:45:01.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.802+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21d0003530 con 0x7f21ec15b1c0 2024-05-27T06:45:01.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21e3e00640 1 -- 172.21.15.81:0/2057156003 >> v1:172.21.15.106:6789/0 conn(0x7f21ec1545e0 legacy=0x7f21ec06e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2108245834 0 0) 0x7f21d0003530 con 0x7f21ec15b1c0 2024-05-27T06:45:01.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21e3e00640 1 -- 172.21.15.81:0/2057156003 >> v1:172.21.15.81:6789/0 conn(0x7f21ec15b1c0 legacy=0x7f21ec0c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21e3e00640 1 -- 172.21.15.81:0/2057156003 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ec0caca0 con 0x7f21ec1549f0 2024-05-27T06:45:01.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21e3e00640 1 --2- 172.21.15.81:0/2057156003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1549f0 0x7f21ec15ac80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d40097c0 tx=0x7f21d401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7c3976e350b21b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:01.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d40b0070 con 0x7f21ec1549f0 2024-05-27T06:45:01.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d40ab400 con 0x7f21ec1549f0 2024-05-27T06:45:01.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.806+0000 7f21ebe00640 1 -- 172.21.15.81:0/2057156003 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4001a80 con 0x7f21ec1549f0 2024-05-27T06:45:01.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- 172.21.15.81:0/2057156003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1549f0 msgr2=0x7f21ec15ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 --2- 172.21.15.81:0/2057156003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1549f0 0x7f21ec15ac80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21d40097c0 tx=0x7f21d401c3c0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- 172.21.15.81:0/2057156003 shutdown_connections 2024-05-27T06:45:01.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 --2- 172.21.15.81:0/2057156003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1549f0 0x7f21ec15ac80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- 172.21.15.81:0/2057156003 >> 172.21.15.81:0/2057156003 conn(0x7f21ec071850 msgr2=0x7f21ec0ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:01.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- 172.21.15.81:0/2057156003 shutdown_connections 2024-05-27T06:45:01.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- 172.21.15.81:0/2057156003 wait complete. 2024-05-27T06:45:01.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 Processor -- start 2024-05-27T06:45:01.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.810+0000 7f21f3c00640 1 -- start start 2024-05-27T06:45:01.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1545e0 0x7f21ec1f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:01.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:01.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f21ec15b1c0 0x7f21ec14c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:01.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21ec0c5e00 con 0x7f21ec1549f0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f21ec0c5c80 con 0x7f21ec1545e0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f21ec0c89a0 con 0x7f21ec15b1c0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47794/0 (socket says 172.21.15.81:47794) 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 -- 172.21.15.81:0/1242610544 learned_addr learned my addr 172.21.15.81:0/1242610544 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f1200640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1545e0 0x7f21ec1f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f1c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f21ec15b1c0 0x7f21ec14c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 -- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f21ec15b1c0 msgr2=0x7f21ec14c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f21ec15b1c0 0x7f21ec14c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 -- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1545e0 msgr2=0x7f21ec1f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1545e0 0x7f21ec1f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 -- 172.21.15.81:0/1242610544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ec0c8110 con 0x7f21ec1549f0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e3e00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f21d40ae040 tx=0x7f21d401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d40bb020 con 0x7f21ec1549f0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21d40b1040 con 0x7f21ec1549f0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d40b0050 con 0x7f21ec1549f0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ec0c6ba0 con 0x7f21ec1549f0 2024-05-27T06:45:01.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ec0cee90 con 0x7f21ec1549f0 2024-05-27T06:45:01.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ec06a560 con 0x7f21ec1549f0 2024-05-27T06:45:01.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.814+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f21d40be070 con 0x7f21ec1549f0 2024-05-27T06:45:01.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.818+0000 7f21e9600640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 0x7f21c8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:01.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.818+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f21d4128790 con 0x7f21ec1549f0 2024-05-27T06:45:01.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.818+0000 7f21f1200640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 0x7f21c8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:01.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.818+0000 7f21f1200640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 0x7f21c8071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21dc007fd0 tx=0x7f21dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:01.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:01.822+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f21d4015fb0 con 0x7f21ec1549f0 2024-05-27T06:45:02.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.014+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f21ec0750a0 con 0x7f21ec1549f0 2024-05-27T06:45:02.617 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 2 and data pool 3 2024-05-27T06:45:02.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.622+0000 7f21e9600640 1 -- 172.21.15.81:0/1242610544 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f21d40f5080 con 0x7f21ec1549f0 2024-05-27T06:45:02.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 msgr2=0x7f21c8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 0x7f21c8071910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21dc007fd0 tx=0x7f21dc009040 comp rx=0 tx=0).stop 2024-05-27T06:45:02.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 msgr2=0x7f21ec0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f21d40ae040 tx=0x7f21d401ac00 comp rx=0 tx=0).stop 2024-05-27T06:45:02.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 shutdown_connections 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f21ec15b1c0 0x7f21ec14c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f21c806f400 0x7f21c8071910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ec1549f0 0x7f21ec0ccb50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 --2- 172.21.15.81:0/1242610544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21ec1545e0 0x7f21ec1f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.626+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 >> 172.21.15.81:0/1242610544 conn(0x7f21ec071850 msgr2=0x7f21ec074110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.630+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 shutdown_connections 2024-05-27T06:45:02.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.630+0000 7f21f3c00640 1 -- 172.21.15.81:0/1242610544 wait complete. 2024-05-27T06:45:02.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 Processor -- start 2024-05-27T06:45:02.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 -- start start 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd60160470 0x7fdd60160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd6006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd600cd390 con 0x7fdd60156900 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fdd6015fd00 con 0x7fdd60160470 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd65800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdd6015ffe0 con 0x7fdd6006c790 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5ea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdd60156900 0x7fdd600cc660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38582/0 (socket says 172.21.15.81:38582) 2024-05-27T06:45:02.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5ea00640 1 -- 172.21.15.81:0/3789833040 learned_addr learned my addr 172.21.15.81:0/3789833040 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5f400640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd6006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5e000640 1 --2- 172.21.15.81:0/3789833040 >> v2:172.21.15.106:3300/0 conn(0x7fdd60160470 0x7fdd60160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404653717 0 0) 0x7fdd600cd390 con 0x7fdd60156900 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.806+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd44003530 con 0x7fdd60156900 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3258789516 0 0) 0x7fdd44003530 con 0x7fdd60156900 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 msgr2=0x7fdd6006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd6006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 >> v2:172.21.15.106:3300/0 conn(0x7fdd60160470 msgr2=0x7fdd60160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 --2- 172.21.15.81:0/3789833040 >> v2:172.21.15.106:3300/0 conn(0x7fdd60160470 0x7fdd60160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6006f120 con 0x7fdd60156900 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fdd50001700 con 0x7fdd60156900 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 >> v1:172.21.15.81:6789/0 conn(0x7fdd60156900 legacy=0x7fdd600cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd44003a10 0x7fdd44003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd44004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44004d50 0x7fdd44005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6015fd00 con 0x7fdd44003a10 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6015ffe0 con 0x7fdd44004d50 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4400bb00 con 0x7fdd6006c790 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd44003a10 0x7fdd44003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5e000640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd44004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5f400640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44004d50 0x7fdd44005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 -- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 msgr2=0x7fdd44004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6006c790 0x7fdd44004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 -- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44004d50 msgr2=0x7fdd44005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:02.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44004d50 0x7fdd44005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 -- 172.21.15.81:0/3789833040 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd44003530 con 0x7fdd44003a10 2024-05-27T06:45:02.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.810+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd44003a10 0x7fdd44003e80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000c210 tx=0x7fdd5001f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e450ed716aa204 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:02.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50007d40 con 0x7fdd44003a10 2024-05-27T06:45:02.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd50005b00 con 0x7fdd44003a10 2024-05-27T06:45:02.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd5d600640 1 -- 172.21.15.81:0/3789833040 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd50007070 con 0x7fdd44003a10 2024-05-27T06:45:02.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- 172.21.15.81:0/3789833040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd44003a10 msgr2=0x7fdd44003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 --2- 172.21.15.81:0/3789833040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd44003a10 0x7fdd44003e80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000c210 tx=0x7fdd5001f3e0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd5f400640 1 -- 172.21.15.81:0/3789833040 reap_dead start 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- 172.21.15.81:0/3789833040 shutdown_connections 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- 172.21.15.81:0/3789833040 >> 172.21.15.81:0/3789833040 conn(0x7fdd600cb2e0 msgr2=0x7fdd600cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- 172.21.15.81:0/3789833040 shutdown_connections 2024-05-27T06:45:02.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- 172.21.15.81:0/3789833040 wait complete. 2024-05-27T06:45:02.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 Processor -- start 2024-05-27T06:45:02.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.814+0000 7fdd65800640 1 -- start start 2024-05-27T06:45:02.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44003a10 0x7fdd601f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6015b110 0x7fdd6015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6006f9c0 con 0x7fdd6015cf30 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6006f6e0 con 0x7fdd44003a10 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdd6006d5b0 con 0x7fdd6015b110 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47802/0 (socket says 172.21.15.81:47802) 2024-05-27T06:45:02.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 -- 172.21.15.81:0/2121211939 learned_addr learned my addr 172.21.15.81:0/2121211939 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5f400640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6015b110 0x7fdd6015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44003a10 0x7fdd601f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 -- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6015b110 msgr2=0x7fdd6015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6015b110 0x7fdd6015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 -- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44003a10 msgr2=0x7fdd601f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44003a10 0x7fdd601f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 -- 172.21.15.81:0/2121211939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd600c6540 con 0x7fdd6015cf30 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd5e000640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006cf0 tx=0x7fdd4c011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c019070 con 0x7fdd6015cf30 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4c005bb0 con 0x7fdd6015cf30 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd601e9ae0 con 0x7fdd6015cf30 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c00f8e0 con 0x7fdd6015cf30 2024-05-27T06:45:02.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd601f2970 con 0x7fdd6015cf30 2024-05-27T06:45:02.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.818+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd6006a560 con 0x7fdd6015cf30 2024-05-27T06:45:02.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.822+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd4c01c070 con 0x7fdd6015cf30 2024-05-27T06:45:02.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.822+0000 7fdd56a00640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 0x7fdd40071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:02.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.822+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdd4c08f7b0 con 0x7fdd6015cf30 2024-05-27T06:45:02.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.822+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 0x7fdd40071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:02.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.822+0000 7fdd5ea00640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 0x7fdd40071940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000c210 tx=0x7fdd6015d7a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:02.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.826+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdd4c05c600 con 0x7fdd6015cf30 2024-05-27T06:45:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd600ccf40 con 0x7fdd6015cf30 2024-05-27T06:45:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd56a00640 1 -- 172.21.15.81:0/2121211939 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16606 (secure 0 0 0) 0x7fdd4c05bf00 con 0x7fdd6015cf30 2024-05-27T06:45:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:02.981 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:02.016872+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:02.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 msgr2=0x7fdd40071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 0x7fdd40071940 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdd5000c210 tx=0x7fdd6015d7a0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 msgr2=0x7fdd6015abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c006cf0 tx=0x7fdd4c011dc0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 shutdown_connections 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd6015b110 0x7fdd6015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd4006f430 0x7fdd40071940 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd6015cf30 0x7fdd6015abd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 --2- 172.21.15.81:0/2121211939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd44003a10 0x7fdd601f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 >> 172.21.15.81:0/2121211939 conn(0x7fdd600cb2e0 msgr2=0x7fdd600cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:02.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 shutdown_connections 2024-05-27T06:45:02.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:02.986+0000 7fdd65800640 1 -- 172.21.15.81:0/2121211939 wait complete. 2024-05-27T06:45:03.003 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:03.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 Processor -- start 2024-05-27T06:45:03.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 -- start start 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 0x7f15580d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1558154200 0x7f1558154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15580c78b0 con 0x7f15580ca770 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1558154b50 con 0x7f1558154200 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155f800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f15580c6ee0 con 0x7f15580cfdb0 2024-05-27T06:45:03.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1558154200 0x7f1558154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1558154200 0x7f1558154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47812/0 (socket says 172.21.15.81:47812) 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 -- 172.21.15.81:0/3180794534 learned_addr learned my addr 172.21.15.81:0/3180794534 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f154fe00640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 0x7f15580d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 -- 172.21.15.81:0/3180794534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 msgr2=0x7f15580d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 0x7f15580d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 -- 172.21.15.81:0/3180794534 >> v1:172.21.15.106:6789/0 conn(0x7f15580ca770 legacy=0x7f15580cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f155d800640 1 -- 172.21.15.81:0/3180794534 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15580c71c0 con 0x7f1558154200 2024-05-27T06:45:03.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.158+0000 7f154fe00640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 0x7f15580d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:03.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155d800640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1558154200 0x7f1558154610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05f3b3b76d9f7ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f1557e00640 1 -- 172.21.15.81:0/3180794534 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15500a5810 con 0x7f1558154200 2024-05-27T06:45:03.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f1557e00640 1 -- 172.21.15.81:0/3180794534 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15500a59b0 con 0x7f1558154200 2024-05-27T06:45:03.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f1557e00640 1 -- 172.21.15.81:0/3180794534 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15500a5d80 con 0x7f1558154200 2024-05-27T06:45:03.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 -- 172.21.15.81:0/3180794534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1558154200 msgr2=0x7f1558154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1558154200 0x7f1558154610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1550009e50 tx=0x7f155001ca30 comp rx=0 tx=0).stop 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 -- 172.21.15.81:0/3180794534 shutdown_connections 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1558154200 0x7f1558154610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 --2- 172.21.15.81:0/3180794534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15580cfdb0 0x7f15580d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 -- 172.21.15.81:0/3180794534 >> 172.21.15.81:0/3180794534 conn(0x7f1558068ae0 msgr2=0x7f1558068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 -- 172.21.15.81:0/3180794534 shutdown_connections 2024-05-27T06:45:03.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.162+0000 7f155f800640 1 -- 172.21.15.81:0/3180794534 wait complete. 2024-05-27T06:45:03.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 Processor -- start 2024-05-27T06:45:03.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 -- start start 2024-05-27T06:45:03.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 0x7f1558146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1558154200 0x7f155814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15580cc050 con 0x7f15580ca770 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f15580cbed0 con 0x7f15580cfdb0 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155f800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1558167590 con 0x7f1558154200 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47818/0 (socket says 172.21.15.81:47818) 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155ce00640 1 -- 172.21.15.81:0/1977804056 learned_addr learned my addr 172.21.15.81:0/1977804056 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f155d800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1558154200 0x7f155814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.166+0000 7f154fe00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 0x7f1558146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 -- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1558154200 msgr2=0x7f155814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1558154200 0x7f155814e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 -- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 msgr2=0x7f1558146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 0x7f1558146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 -- 172.21.15.81:0/1977804056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155806e6f0 con 0x7f15580ca770 2024-05-27T06:45:03.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f154fe00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 0x7f1558146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:03.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155ce00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f154800c3c0 tx=0x7f154800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1548017070 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15480052d0 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1548003820 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15581f6820 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15581f6ae0 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155806a560 con 0x7f15580ca770 2024-05-27T06:45:03.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f154801a070 con 0x7f15580ca770 2024-05-27T06:45:03.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 0x7f1534071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f154808e5a0 con 0x7f15580ca770 2024-05-27T06:45:03.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.170+0000 7f154fe00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 0x7f1534071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.174+0000 7f154fe00640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 0x7f1534071940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f15580c6150 tx=0x7f1540008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.178+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1548005480 con 0x7f15580ca770 2024-05-27T06:45:03.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.346+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1558072280 con 0x7f15580ca770 2024-05-27T06:45:03.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.346+0000 7f1555600640 1 -- 172.21.15.81:0/1977804056 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16606 (secure 0 0 0) 0x7f154805ada0 con 0x7f15580ca770 2024-05-27T06:45:03.342 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:03.342 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:02.016872+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 msgr2=0x7f1534071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 0x7f1534071940 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f15580c6150 tx=0x7f1540008040 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 msgr2=0x7f155806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f154800c3c0 tx=0x7f154800c880 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 shutdown_connections 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1558154200 0x7f155814e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f153406f430 0x7f1534071940 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15580cfdb0 0x7f1558146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 --2- 172.21.15.81:0/1977804056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580ca770 0x7f155806e1b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 >> 172.21.15.81:0/1977804056 conn(0x7f1558068ae0 msgr2=0x7f15581580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 shutdown_connections 2024-05-27T06:45:03.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.350+0000 7f155f800640 1 -- 172.21.15.81:0/1977804056 wait complete. 2024-05-27T06:45:03.365 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-27T06:45:03.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 Processor -- start 2024-05-27T06:45:03.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 -- start start 2024-05-27T06:45:03.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f172815fe30 0x7f17280ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1728074aa0 0x7f1728074eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1728157030 con 0x7f172806e9b0 2024-05-27T06:45:03.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1728160590 con 0x7f172815fe30 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f1730000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1728160870 con 0x7f1728074aa0 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f172815fe30 0x7f17280ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f172815fe30 0x7f17280ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47828/0 (socket says 172.21.15.81:47828) 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172e000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f172806e9b0 0x7f172806edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38596/0 (socket says 172.21.15.81:38596) 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 -- 172.21.15.81:0/1238394296 learned_addr learned my addr 172.21.15.81:0/1238394296 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172cc00640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1728074aa0 0x7f1728074eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 -- 172.21.15.81:0/1238394296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1728074aa0 msgr2=0x7f1728074eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1728074aa0 0x7f1728074eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 -- 172.21.15.81:0/1238394296 >> v1:172.21.15.81:6789/0 conn(0x7f172806e9b0 legacy=0x7f172806edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.522+0000 7f172d600640 1 -- 172.21.15.81:0/1238394296 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1728160b50 con 0x7f172815fe30 2024-05-27T06:45:03.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f172d600640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172815fe30 0x7f17280ca4f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f171c009870 tx=0x7f171c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8569ebe7c457fbcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1727e00640 1 -- 172.21.15.81:0/1238394296 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c0b5070 con 0x7f172815fe30 2024-05-27T06:45:03.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1727e00640 1 -- 172.21.15.81:0/1238394296 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171c0a8a50 con 0x7f172815fe30 2024-05-27T06:45:03.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1727e00640 1 -- 172.21.15.81:0/1238394296 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c0a8e20 con 0x7f172815fe30 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 -- 172.21.15.81:0/1238394296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172815fe30 msgr2=0x7f17280ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172815fe30 0x7f17280ca4f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f171c009870 tx=0x7f171c01ed50 comp rx=0 tx=0).stop 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 -- 172.21.15.81:0/1238394296 shutdown_connections 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1728074aa0 0x7f1728074eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 --2- 172.21.15.81:0/1238394296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f172815fe30 0x7f17280ca4f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 -- 172.21.15.81:0/1238394296 >> 172.21.15.81:0/1238394296 conn(0x7f1728072500 msgr2=0x7f1728072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 -- 172.21.15.81:0/1238394296 shutdown_connections 2024-05-27T06:45:03.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.526+0000 7f1730000640 1 -- 172.21.15.81:0/1238394296 wait complete. 2024-05-27T06:45:03.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 Processor -- start 2024-05-27T06:45:03.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 -- start start 2024-05-27T06:45:03.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f172806e9b0 0x7f1728157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f172815fe30 0x7f172814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17280caa30 con 0x7f1728074aa0 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1728160240 con 0x7f172815fe30 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f1730000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f17280cabb0 con 0x7f172806e9b0 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f172cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f172cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47840/0 (socket says 172.21.15.81:47840) 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f172cc00640 1 -- 172.21.15.81:0/3124796674 learned_addr learned my addr 172.21.15.81:0/3124796674 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f172e000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f172815fe30 0x7f172814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.530+0000 7f172d600640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f172806e9b0 0x7f1728157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 -- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f172806e9b0 msgr2=0x7f1728157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f172806e9b0 0x7f1728157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 -- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f172815fe30 msgr2=0x7f172814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f172815fe30 0x7f172814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 -- 172.21.15.81:0/3124796674 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17281e8980 con 0x7f1728074aa0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f172cc00640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f171400d5d0 tx=0x7f171400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714004240 con 0x7f1728074aa0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17140043e0 con 0x7f1728074aa0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1714003840 con 0x7f1728074aa0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17281e78e0 con 0x7f1728074aa0 2024-05-27T06:45:03.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17281e7ea0 con 0x7f1728074aa0 2024-05-27T06:45:03.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f172806a560 con 0x7f1728074aa0 2024-05-27T06:45:03.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.534+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f17140039e0 con 0x7f1728074aa0 2024-05-27T06:45:03.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.538+0000 7f1725600640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 0x7f1708071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:03.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.538+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f171408f5d0 con 0x7f1728074aa0 2024-05-27T06:45:03.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.538+0000 7f172d600640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 0x7f1708071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:03.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.538+0000 7f172d600640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 0x7f1708071910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b2000 tx=0x7f171c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:03.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.542+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f171405c420 con 0x7f1728074aa0 2024-05-27T06:45:03.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:03.734+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f17280c52a0 con 0x7f1728074aa0 2024-05-27T06:45:04.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.622+0000 7f1725600640 1 -- 172.21.15.81:0/3124796674 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f171405bd20 con 0x7f1728074aa0 2024-05-27T06:45:04.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 msgr2=0x7f1708071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 0x7f1708071910 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b2000 tx=0x7f171c008040 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 msgr2=0x7f172814df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f171400d5d0 tx=0x7f171400da90 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 shutdown_connections 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f170806f400 0x7f1708071910 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f172815fe30 0x7f172814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1728074aa0 0x7f172814df20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 --2- 172.21.15.81:0/3124796674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f172806e9b0 0x7f1728157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 >> 172.21.15.81:0/3124796674 conn(0x7f1728072500 msgr2=0x7f172815e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 shutdown_connections 2024-05-27T06:45:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.626+0000 7f1730000640 1 -- 172.21.15.81:0/3124796674 wait complete. 2024-05-27T06:45:04.642 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-05-27T06:45:04.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-05-27T06:45:04.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 Processor -- start 2024-05-27T06:45:04.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 -- start start 2024-05-27T06:45:04.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb8815a230 0x7ffb88073e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:04.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb880c67f0 con 0x7ffb881595d0 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb880726a0 con 0x7ffb880c6bd0 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb8e800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ffb88072450 con 0x7ffb8815a230 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb86a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb8815a230 0x7ffb88073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb86a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb8815a230 0x7ffb88073e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47852/0 (socket says 172.21.15.81:47852) 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb86a00640 1 -- 172.21.15.81:0/1613959651 learned_addr learned my addr 172.21.15.81:0/1613959651 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.802+0000 7ffb87400640 1 --1- 172.21.15.81:0/1613959651 >> v1:172.21.15.81:6789/0 conn(0x7ffb881595d0 0x7ffb880c77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38602/0 (socket says 172.21.15.81:38602) 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86a00640 1 -- 172.21.15.81:0/1613959651 >> v1:172.21.15.106:6789/0 conn(0x7ffb880c6bd0 legacy=0x7ffb880743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86a00640 1 -- 172.21.15.81:0/1613959651 >> v1:172.21.15.81:6789/0 conn(0x7ffb881595d0 legacy=0x7ffb880c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86a00640 1 -- 172.21.15.81:0/1613959651 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb880c5080 con 0x7ffb8815a230 2024-05-27T06:45:04.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86a00640 1 --2- 172.21.15.81:0/1613959651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb88073e90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009870 tx=0x7ffb7401e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4329bf79470d8f77 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:04.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86000640 1 -- 172.21.15.81:0/1613959651 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb740a8660 con 0x7ffb8815a230 2024-05-27T06:45:04.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86000640 1 -- 172.21.15.81:0/1613959651 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb740a8800 con 0x7ffb8815a230 2024-05-27T06:45:04.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.806+0000 7ffb86000640 1 -- 172.21.15.81:0/1613959651 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb740a8bd0 con 0x7ffb8815a230 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- 172.21.15.81:0/1613959651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 msgr2=0x7ffb88073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1613959651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb88073e90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009870 tx=0x7ffb7401e770 comp rx=0 tx=0).stop 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- 172.21.15.81:0/1613959651 shutdown_connections 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1613959651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb88073e90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- 172.21.15.81:0/1613959651 >> 172.21.15.81:0/1613959651 conn(0x7ffb88070760 msgr2=0x7ffb880cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- 172.21.15.81:0/1613959651 shutdown_connections 2024-05-27T06:45:04.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- 172.21.15.81:0/1613959651 wait complete. 2024-05-27T06:45:04.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 Processor -- start 2024-05-27T06:45:04.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.810+0000 7ffb8e800640 1 -- start start 2024-05-27T06:45:04.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb880c6bd0 0x7ffb880c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:04.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 0x7ffb88144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:04.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8806d330 con 0x7ffb8815a230 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ffb8806d1b0 con 0x7ffb881595d0 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ffb8806d4b0 con 0x7ffb880c6bd0 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47868/0 (socket says 172.21.15.81:47868) 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 -- 172.21.15.81:0/1272228887 learned_addr learned my addr 172.21.15.81:0/1272228887 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb86a00640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 0x7ffb88144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87400640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb880c6bd0 0x7ffb880c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 -- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb880c6bd0 msgr2=0x7ffb880c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb880c6bd0 0x7ffb880c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 -- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 msgr2=0x7ffb88144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:04.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 0x7ffb88144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:04.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 -- 172.21.15.81:0/1272228887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb880c81c0 con 0x7ffb8815a230 2024-05-27T06:45:04.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb86a00640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 0x7ffb88144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:04.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb87e00640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000c3c0 tx=0x7ffb8000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:04.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb80017070 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb800052d0 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb800054c0 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb881f5b40 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb881434b0 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8806a560 con 0x7ffb8815a230 2024-05-27T06:45:04.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.814+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffb8001a070 con 0x7ffb8815a230 2024-05-27T06:45:04.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.818+0000 7ffb7f400640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 0x7ffb68071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:04.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.818+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffb8008e5c0 con 0x7ffb8815a230 2024-05-27T06:45:04.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.818+0000 7ffb87400640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 0x7ffb68071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:04.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.818+0000 7ffb87400640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 0x7ffb68071910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb780012f0 tx=0x7ffb78005b80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:04.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:04.822+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ffb8005b600 con 0x7ffb8815a230 2024-05-27T06:45:05.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.014+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7ffb88077070 con 0x7ffb8815a230 2024-05-27T06:45:05.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.626+0000 7ffb7f400640 1 -- 172.21.15.81:0/1272228887 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7ffb8005af00 con 0x7ffb8815a230 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 msgr2=0x7ffb68071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 0x7ffb68071910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb780012f0 tx=0x7ffb78005b80 comp rx=0 tx=0).stop 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 msgr2=0x7ffb8814e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000c3c0 tx=0x7ffb8000c880 comp rx=0 tx=0).stop 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 shutdown_connections 2024-05-27T06:45:05.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb6806f400 0x7ffb68071910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb8815a230 0x7ffb8814e930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb881595d0 0x7ffb88144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 --2- 172.21.15.81:0/1272228887 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffb880c6bd0 0x7ffb880c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 >> 172.21.15.81:0/1272228887 conn(0x7ffb88070760 msgr2=0x7ffb8814b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 shutdown_connections 2024-05-27T06:45:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.630+0000 7ffb8e800640 1 -- 172.21.15.81:0/1272228887 wait complete. 2024-05-27T06:45:05.647 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-27T06:45:05.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 Processor -- start 2024-05-27T06:45:05.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 -- start start 2024-05-27T06:45:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5ccc0c6bd0 0x7f5ccc0745a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ccc0c67f0 con 0x7f5ccc15a230 2024-05-27T06:45:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5ccc072450 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd4600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc0c53f0 con 0x7f5ccc1595d0 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd1c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd2600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5ccc0c6bd0 0x7f5ccc0745a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd1200640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f5ccc15a230 0x7f5ccc073e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59028/0 (socket says 172.21.15.81:59028) 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd1200640 1 -- 172.21.15.81:0/880666669 learned_addr learned my addr 172.21.15.81:0/880666669 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5ccbe00640 1 -- 172.21.15.81:0/880666669 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3996117385 0 0) 0x7f5ccc0c67f0 con 0x7f5ccc15a230 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5ccbe00640 1 -- 172.21.15.81:0/880666669 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb4003530 con 0x7f5ccc15a230 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd2600640 1 -- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 msgr2=0x7f5ccc0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd2600640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc0c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd2600640 1 -- 172.21.15.81:0/880666669 >> v1:172.21.15.106:6789/0 conn(0x7f5ccc15a230 legacy=0x7f5ccc073e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd2600640 1 -- 172.21.15.81:0/880666669 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc0726a0 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.806+0000 7f5cd1c00640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd2600640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc0745a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009d90 tx=0x7f5cc401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570473d06e915dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5ccbe00640 1 -- 172.21.15.81:0/880666669 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40b2070 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5ccbe00640 1 -- 172.21.15.81:0/880666669 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc40a5a50 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5ccbe00640 1 -- 172.21.15.81:0/880666669 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40a5e00 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 -- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc0c6bd0 msgr2=0x7f5ccc0745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc0745a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009d90 tx=0x7f5cc401c970 comp rx=0 tx=0).stop 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 -- 172.21.15.81:0/880666669 shutdown_connections 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc0745a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 --2- 172.21.15.81:0/880666669 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 -- 172.21.15.81:0/880666669 >> 172.21.15.81:0/880666669 conn(0x7f5ccc070760 msgr2=0x7f5ccc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:05.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 -- 172.21.15.81:0/880666669 shutdown_connections 2024-05-27T06:45:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.810+0000 7f5cd4600640 1 -- 172.21.15.81:0/880666669 wait complete. 2024-05-27T06:45:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 Processor -- start 2024-05-27T06:45:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 -- start start 2024-05-27T06:45:05.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc15a230 0x7f5ccc146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc166690 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc072980 con 0x7f5ccc15a230 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd4600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc166810 con 0x7f5ccc1595d0 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57068/0 (socket says 172.21.15.81:57068) 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 -- 172.21.15.81:0/806860677 learned_addr learned my addr 172.21.15.81:0/806860677 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd2600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc15a230 0x7f5ccc146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1200640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 -- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 msgr2=0x7f5ccc14c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc14c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 -- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc15a230 msgr2=0x7f5ccc146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc15a230 0x7f5ccc146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1c00640 1 -- 172.21.15.81:0/806860677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc148a90 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.814+0000 7f5cd1200640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cd1c00640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000e980 tx=0x7f5cc000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0017070 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc000c9a0 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc000cd70 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc148d70 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc0c9840 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ccc06a560 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5cc001a070 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 0x7f5cac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:05.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5cc008f430 con 0x7f5ccc0c6bd0 2024-05-27T06:45:05.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.818+0000 7f5cd1200640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 0x7f5cac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:05.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.822+0000 7f5cd1200640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 0x7f5cac071940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc074180 tx=0x7f5cb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:05.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:05.826+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5cc005bcc0 con 0x7f5ccc0c6bd0 2024-05-27T06:45:06.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.014+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5ccc0c4260 con 0x7f5ccc0c6bd0 2024-05-27T06:45:06.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.630+0000 7f5cc9600640 1 -- 172.21.15.81:0/806860677 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f5cc005ffc0 con 0x7f5ccc0c6bd0 2024-05-27T06:45:06.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 msgr2=0x7f5cac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 0x7f5cac071940 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc074180 tx=0x7f5cb8009040 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 msgr2=0x7f5ccc1f96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000e980 tx=0x7f5cc000ee40 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 shutdown_connections 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5ccc1595d0 0x7f5ccc14c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5cac06f430 0x7f5cac071940 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ccc15a230 0x7f5ccc146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 --2- 172.21.15.81:0/806860677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ccc0c6bd0 0x7f5ccc1f96c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 >> 172.21.15.81:0/806860677 conn(0x7f5ccc070760 msgr2=0x7f5ccc074310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:06.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 shutdown_connections 2024-05-27T06:45:06.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.634+0000 7f5cd4600640 1 -- 172.21.15.81:0/806860677 wait complete. 2024-05-27T06:45:06.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-27T06:45:06.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 Processor -- start 2024-05-27T06:45:06.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 -- start start 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd584074ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd584073ac0 con 0x7fd5840c73e0 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5840ce080 con 0x7fd5840c6fd0 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd58b400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd584154d20 con 0x7fd584154910 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd589400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd5840c73e0 0x7fd584164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:36996/0 (socket says 172.21.15.81:36996) 2024-05-27T06:45:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd589400640 1 -- 172.21.15.81:0/179137524 learned_addr learned my addr 172.21.15.81:0/179137524 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd588a00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd584074ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241439137 0 0) 0x7fd584073ac0 con 0x7fd5840c73e0 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd56c003530 con 0x7fd5840c73e0 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950751932 0 0) 0x7fd5840ce080 con 0x7fd5840c6fd0 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd584073ac0 con 0x7fd5840c6fd0 2024-05-27T06:45:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3914837419 0 0) 0x7fd56c003530 con 0x7fd5840c73e0 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 msgr2=0x7fd584074ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd584074ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 >> v1:172.21.15.106:6789/0 conn(0x7fd5840c6fd0 legacy=0x7fd58415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5840c8780 con 0x7fd5840c73e0 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fd57c001a80 con 0x7fd5840c73e0 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 >> v1:172.21.15.81:6789/0 conn(0x7fd5840c73e0 legacy=0x7fd584164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c003c70 0x7fd56c0040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd56c004810 0x7fd56c004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd56c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd584154d20 con 0x7fd56c004810 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd56c005530 con 0x7fd56c003c70 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd56c00bf10 con 0x7fd584154910 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd583e00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd56c004810 0x7fd56c004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd588a00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c003c70 0x7fd56c0040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.810+0000 7fd589400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd56c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 -- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 msgr2=0x7fd56c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154910 0x7fd56c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 -- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c003c70 msgr2=0x7fd56c0040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c003c70 0x7fd56c0040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 -- 172.21.15.81:0/179137524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56c003530 con 0x7fd56c004810 2024-05-27T06:45:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd588a00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c003c70 0x7fd56c0040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:06.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583e00640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd56c004810 0x7fd56c004c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd5840ce900 tx=0x7fd57001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26554a6fb9bf3d59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:06.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd570001550 con 0x7fd56c004810 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5700052d0 con 0x7fd56c004810 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.814+0000 7fd583400640 1 -- 172.21.15.81:0/179137524 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd570007260 con 0x7fd56c004810 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- 172.21.15.81:0/179137524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd56c004810 msgr2=0x7fd56c004c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 --2- 172.21.15.81:0/179137524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd56c004810 0x7fd56c004c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd5840ce900 tx=0x7fd57001ec30 comp rx=0 tx=0).stop 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd589400640 1 -- 172.21.15.81:0/179137524 reap_dead start 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- 172.21.15.81:0/179137524 shutdown_connections 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- 172.21.15.81:0/179137524 >> 172.21.15.81:0/179137524 conn(0x7fd584070760 msgr2=0x7fd5840cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- 172.21.15.81:0/179137524 shutdown_connections 2024-05-27T06:45:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- 172.21.15.81:0/179137524 wait complete. 2024-05-27T06:45:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 Processor -- start 2024-05-27T06:45:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- start start 2024-05-27T06:45:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c004810 0x7fd584155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154ed0 0x7fd5840ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5840732c0 con 0x7fd5841f0a40 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd584072fe0 con 0x7fd56c004810 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.818+0000 7fd58b400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd5840735a0 con 0x7fd584154ed0 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd588a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c004810 0x7fd584155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57088/0 (socket says 172.21.15.81:57088) 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 -- 172.21.15.81:0/3830327375 learned_addr learned my addr 172.21.15.81:0/3830327375 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd583e00640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154ed0 0x7fd5840ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 -- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154ed0 msgr2=0x7fd5840ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154ed0 0x7fd5840ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 -- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c004810 msgr2=0x7fd584155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c004810 0x7fd584155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 -- 172.21.15.81:0/3830327375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd584157f40 con 0x7fd5841f0a40 2024-05-27T06:45:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd589400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd584072550 tx=0x7fd57c007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5780096b0 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c019070 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd584157c60 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00ac10 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd584153760 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd58406a560 con 0x7fd5841f0a40 2024-05-27T06:45:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.822+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd57c01e460 con 0x7fd5841f0a40 2024-05-27T06:45:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.826+0000 7fd580c00640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 0x7fd564071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.826+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd57c08f5d0 con 0x7fd5841f0a40 2024-05-27T06:45:06.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.826+0000 7fd588a00640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 0x7fd564071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:06.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.826+0000 7fd588a00640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 0x7fd564071910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd578009c60 tx=0x7fd578009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:06.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:06.830+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd57c05c420 con 0x7fd5841f0a40 2024-05-27T06:45:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.022+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd58414f260 con 0x7fd5841f0a40 2024-05-27T06:45:07.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.646+0000 7fd580c00640 1 -- 172.21.15.81:0/3830327375 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7fd57c05bd20 con 0x7fd5841f0a40 2024-05-27T06:45:07.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 msgr2=0x7fd564071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 0x7fd564071910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd578009c60 tx=0x7fd578009190 comp rx=0 tx=0).stop 2024-05-27T06:45:07.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 msgr2=0x7fd5841489d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd584072550 tx=0x7fd57c007b10 comp rx=0 tx=0).stop 2024-05-27T06:45:07.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 shutdown_connections 2024-05-27T06:45:07.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd584154ed0 0x7fd5840ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd56406f400 0x7fd564071910 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5841f0a40 0x7fd5841489d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 --2- 172.21.15.81:0/3830327375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd56c004810 0x7fd584155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 >> 172.21.15.81:0/3830327375 conn(0x7fd584070760 msgr2=0x7fd58414eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 shutdown_connections 2024-05-27T06:45:07.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.650+0000 7fd58b400640 1 -- 172.21.15.81:0/3830327375 wait complete. 2024-05-27T06:45:07.666 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-27T06:45:07.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.822+0000 7fceaa000640 1 Processor -- start 2024-05-27T06:45:07.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.822+0000 7fceaa000640 1 -- start start 2024-05-27T06:45:07.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fceaa000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcea41546f0 0x7fcea4072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fceaa000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea415a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fceaa000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea415db00 con 0x7fcea40732f0 2024-05-27T06:45:07.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fceaa000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fcea415e6f0 con 0x7fcea41546f0 2024-05-27T06:45:07.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fceaa000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcea40755b0 con 0x7fcea415a510 2024-05-27T06:45:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcea41546f0 0x7fcea4072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcea41546f0 0x7fcea4072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57092/0 (socket says 172.21.15.81:57092) 2024-05-27T06:45:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 -- 172.21.15.81:0/936739736 learned_addr learned my addr 172.21.15.81:0/936739736 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3e00640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 -- 172.21.15.81:0/936739736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 msgr2=0x7fcea415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 -- 172.21.15.81:0/936739736 >> v1:172.21.15.106:6789/0 conn(0x7fcea40732f0 legacy=0x7fcea40c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea3400640 1 -- 172.21.15.81:0/936739736 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea415ae60 con 0x7fcea41546f0 2024-05-27T06:45:07.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.826+0000 7fcea2a00640 1 -- 172.21.15.81:0/936739736 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897464290 0 0) 0x7fcea415db00 con 0x7fcea40732f0 2024-05-27T06:45:07.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fcea3400640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea4072db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce94009e50 tx=0x7fce9401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c191e6319d02226 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fcea2a00640 1 -- 172.21.15.81:0/936739736 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce940b4070 con 0x7fcea41546f0 2024-05-27T06:45:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fcea2a00640 1 -- 172.21.15.81:0/936739736 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce940af400 con 0x7fcea41546f0 2024-05-27T06:45:07.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fcea2a00640 1 -- 172.21.15.81:0/936739736 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce940af7d0 con 0x7fcea41546f0 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 -- 172.21.15.81:0/936739736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 msgr2=0x7fcea4072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea4072db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce94009e50 tx=0x7fce9401f210 comp rx=0 tx=0).stop 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 -- 172.21.15.81:0/936739736 shutdown_connections 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 --2- 172.21.15.81:0/936739736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea4072db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 -- 172.21.15.81:0/936739736 >> 172.21.15.81:0/936739736 conn(0x7fcea4070700 msgr2=0x7fcea40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 -- 172.21.15.81:0/936739736 shutdown_connections 2024-05-27T06:45:07.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.830+0000 7fceaa000640 1 -- 172.21.15.81:0/936739736 wait complete. 2024-05-27T06:45:07.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 Processor -- start 2024-05-27T06:45:07.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- start start 2024-05-27T06:45:07.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcea40732f0 0x7fcea4151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea4148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcea415e0a0 con 0x7fcea41546f0 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcea415ddc0 con 0x7fcea40732f0 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcea415e380 con 0x7fcea415a510 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57104/0 (socket says 172.21.15.81:57104) 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 -- 172.21.15.81:0/3123935567 learned_addr learned my addr 172.21.15.81:0/3123935567 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fcea3400640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcea40732f0 0x7fcea4151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fcea3e00640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea4148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 -- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 msgr2=0x7fcea4148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea4148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 -- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcea40732f0 msgr2=0x7fcea4151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcea40732f0 0x7fcea4151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:07.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 -- 172.21.15.81:0/3123935567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea41f5280 con 0x7fcea41546f0 2024-05-27T06:45:07.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9aa00640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c370 tx=0x7fce8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:07.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c017070 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce8c003600 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea4153600 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c003a50 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.834+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea4153bc0 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea406a560 con 0x7fcea41546f0 2024-05-27T06:45:07.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fce8c01a070 con 0x7fcea41546f0 2024-05-27T06:45:07.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fce9be00640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 0x7fce80071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:07.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fce8c08e590 con 0x7fcea41546f0 2024-05-27T06:45:07.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fcea3400640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 0x7fce80071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:07.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.838+0000 7fcea3400640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 0x7fce80071940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fce940b1000 tx=0x7fce940aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:07.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:07.842+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fce8c05ae20 con 0x7fcea41546f0 2024-05-27T06:45:08.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcea40c8e20 con 0x7fcea41546f0 2024-05-27T06:45:08.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fce9be00640 1 -- 172.21.15.81:0/3123935567 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6813 (secure 0 0 0) 0x7fce8c05f120 con 0x7fcea41546f0 2024-05-27T06:45:08.028 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:08.028 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:07.024828+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-05-27T06:45:08.028 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 msgr2=0x7fce80071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 0x7fce80071940 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fce940b1000 tx=0x7fce940aa000 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 msgr2=0x7fcea41f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c370 tx=0x7fce8c00c830 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 shutdown_connections 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcea415a510 0x7fcea4148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce8006f430 0x7fce80071940 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea41546f0 0x7fcea41f4d40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 --2- 172.21.15.81:0/3123935567 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcea40732f0 0x7fcea4151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 >> 172.21.15.81:0/3123935567 conn(0x7fcea4070700 msgr2=0x7fcea4150bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 shutdown_connections 2024-05-27T06:45:08.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.034+0000 7fceaa000640 1 -- 172.21.15.81:0/3123935567 wait complete. 2024-05-27T06:45:08.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:08.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 Processor -- start 2024-05-27T06:45:08.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 -- start start 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff5c81546f0 0x7ff5c8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff5c815a510 0x7ff5c815a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5c815db00 con 0x7ff5c80732f0 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff5c815e6f0 con 0x7ff5c81546f0 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cfa00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff5c80755b0 con 0x7ff5c815a510 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff5c81546f0 0x7ff5c8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff5c81546f0 0x7ff5c8072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57116/0 (socket says 172.21.15.81:57116) 2024-05-27T06:45:08.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 -- 172.21.15.81:0/1588052039 learned_addr learned my addr 172.21.15.81:0/1588052039 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:08.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cda00640 1 --2- 172.21.15.81:0/1588052039 >> v2:172.21.15.106:3300/0 conn(0x7ff5c815a510 0x7ff5c815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 -- 172.21.15.81:0/1588052039 >> v2:172.21.15.106:3300/0 conn(0x7ff5c815a510 msgr2=0x7ff5c815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:08.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 --2- 172.21.15.81:0/1588052039 >> v2:172.21.15.106:3300/0 conn(0x7ff5c815a510 0x7ff5c815a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 -- 172.21.15.81:0/1588052039 >> v1:172.21.15.106:6789/0 conn(0x7ff5c80732f0 legacy=0x7ff5c80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.202+0000 7ff5cd000640 1 -- 172.21.15.81:0/1588052039 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c815ae60 con 0x7ff5c81546f0 2024-05-27T06:45:08.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cd000640 1 --2- 172.21.15.81:0/1588052039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c81546f0 0x7ff5c8072db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80ca360 tx=0x7ff5bc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dbbab0640eaab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5c7400640 1 -- 172.21.15.81:0/1588052039 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0b5070 con 0x7ff5c81546f0 2024-05-27T06:45:08.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5c7400640 1 -- 172.21.15.81:0/1588052039 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5bc0a8a10 con 0x7ff5c81546f0 2024-05-27T06:45:08.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5c7400640 1 -- 172.21.15.81:0/1588052039 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0a8de0 con 0x7ff5c81546f0 2024-05-27T06:45:08.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 -- 172.21.15.81:0/1588052039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c81546f0 msgr2=0x7ff5c8072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/1588052039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c81546f0 0x7ff5c8072db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80ca360 tx=0x7ff5bc01ed10 comp rx=0 tx=0).stop 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 -- 172.21.15.81:0/1588052039 shutdown_connections 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/1588052039 >> v2:172.21.15.106:3300/0 conn(0x7ff5c815a510 0x7ff5c815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/1588052039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c81546f0 0x7ff5c8072db0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 -- 172.21.15.81:0/1588052039 >> 172.21.15.81:0/1588052039 conn(0x7ff5c8070700 msgr2=0x7ff5c80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 -- 172.21.15.81:0/1588052039 shutdown_connections 2024-05-27T06:45:08.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.206+0000 7ff5cfa00640 1 -- 172.21.15.81:0/1588052039 wait complete. 2024-05-27T06:45:08.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 Processor -- start 2024-05-27T06:45:08.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- start start 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 0x7ff5c814ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c81546f0 0x7ff5c80c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c815e0a0 con 0x7ff5c815a510 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c815ddc0 con 0x7ff5c80732f0 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff5c815e380 con 0x7ff5c81546f0 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57122/0 (socket says 172.21.15.81:57122) 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 -- 172.21.15.81:0/3636299395 learned_addr learned my addr 172.21.15.81:0/3636299395 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:08.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5c7e00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c81546f0 0x7ff5c80c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cd000640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 0x7ff5c814ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 -- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c81546f0 msgr2=0x7ff5c80c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c81546f0 0x7ff5c80c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 -- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 msgr2=0x7ff5c814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 0x7ff5c814ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 -- 172.21.15.81:0/3636299395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c814f120 con 0x7ff5c815a510 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cd000640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 0x7ff5c814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:08.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cda00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0007d40 tx=0x7ff5c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0014420 con 0x7ff5c815a510 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c0019070 con 0x7ff5c815a510 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5c80c4e60 con 0x7ff5c815a510 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0014820 con 0x7ff5c815a510 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.210+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5c815d890 con 0x7ff5c815a510 2024-05-27T06:45:08.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5c80cd0f0 con 0x7ff5c815a510 2024-05-27T06:45:08.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff5c00149c0 con 0x7ff5c815a510 2024-05-27T06:45:08.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5c4c00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 0x7ff5a8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5c008fac0 con 0x7ff5c815a510 2024-05-27T06:45:08.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5cd000640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 0x7ff5a8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.214+0000 7ff5cd000640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 0x7ff5a8071940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2000 tx=0x7ff5bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.218+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff5c005c3e0 con 0x7ff5c815a510 2024-05-27T06:45:08.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5c806a560 con 0x7ff5c815a510 2024-05-27T06:45:08.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5c4c00640 1 -- 172.21.15.81:0/3636299395 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7ff5c00606e0 con 0x7ff5c815a510 2024-05-27T06:45:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:08.373 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:03.608328+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:08.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 msgr2=0x7ff5a8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 0x7ff5a8071940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2000 tx=0x7ff5bc008040 comp rx=0 tx=0).stop 2024-05-27T06:45:08.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 msgr2=0x7ff5c80cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0007d40 tx=0x7ff5c000e4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 shutdown_connections 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5c81546f0 0x7ff5c80c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff5a806f430 0x7ff5a8071940 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5c815a510 0x7ff5c80cb890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 --2- 172.21.15.81:0/3636299395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5c80732f0 0x7ff5c814ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 >> 172.21.15.81:0/3636299395 conn(0x7ff5c8070700 msgr2=0x7ff5c8153be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 shutdown_connections 2024-05-27T06:45:08.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.378+0000 7ff5cfa00640 1 -- 172.21.15.81:0/3636299395 wait complete. 2024-05-27T06:45:08.395 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-27T06:45:08.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:08.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.546+0000 7f18cde00640 1 Processor -- start 2024-05-27T06:45:08.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.546+0000 7f18cde00640 1 -- start start 2024-05-27T06:45:08.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18cde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f18c80c5b60 0x7f18c80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18cde00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c8072840 con 0x7f18c8072430 2024-05-27T06:45:08.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18cde00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c80c5270 con 0x7f18c8161cd0 2024-05-27T06:45:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18cde00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f18c80c5040 con 0x7f18c80c5b60 2024-05-27T06:45:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18c6a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f18c8072430 0x7f18c8159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37006/0 (socket says 172.21.15.81:37006) 2024-05-27T06:45:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18c6a00640 1 -- 172.21.15.81:0/3631066128 learned_addr learned my addr 172.21.15.81:0/3631066128 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18bfe00640 1 --2- 172.21.15.81:0/3631066128 >> v2:172.21.15.106:3300/0 conn(0x7f18c80c5b60 0x7f18c80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18c6000640 1 -- 172.21.15.81:0/3631066128 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234049629 0 0) 0x7f18c8072840 con 0x7f18c8072430 2024-05-27T06:45:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18c6000640 1 -- 172.21.15.81:0/3631066128 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003530 con 0x7f18c8072430 2024-05-27T06:45:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18bfe00640 1 -- 172.21.15.81:0/3631066128 >> v1:172.21.15.106:6789/0 conn(0x7f18c8161cd0 legacy=0x7f18c81640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18bfe00640 1 -- 172.21.15.81:0/3631066128 >> v1:172.21.15.81:6789/0 conn(0x7f18c8072430 legacy=0x7f18c8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.550+0000 7f18bfe00640 1 -- 172.21.15.81:0/3631066128 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c80c7810 con 0x7f18c80c5b60 2024-05-27T06:45:08.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18bfe00640 1 --2- 172.21.15.81:0/3631066128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c80c5b60 0x7f18c80c5f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18b00097c0 tx=0x7f18b001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4881de1d6a349e00 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18c6000640 1 -- 172.21.15.81:0/3631066128 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b00b0070 con 0x7f18c80c5b60 2024-05-27T06:45:08.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18c6000640 1 -- 172.21.15.81:0/3631066128 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b00ab600 con 0x7f18c80c5b60 2024-05-27T06:45:08.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18c6000640 1 -- 172.21.15.81:0/3631066128 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b00ab9d0 con 0x7f18c80c5b60 2024-05-27T06:45:08.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 -- 172.21.15.81:0/3631066128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c80c5b60 msgr2=0x7f18c80c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 --2- 172.21.15.81:0/3631066128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c80c5b60 0x7f18c80c5f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f18b00097c0 tx=0x7f18b001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 -- 172.21.15.81:0/3631066128 shutdown_connections 2024-05-27T06:45:08.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 --2- 172.21.15.81:0/3631066128 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c80c5b60 0x7f18c80c5f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 -- 172.21.15.81:0/3631066128 >> 172.21.15.81:0/3631066128 conn(0x7f18c8070700 msgr2=0x7f18c80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 -- 172.21.15.81:0/3631066128 shutdown_connections 2024-05-27T06:45:08.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.554+0000 7f18cde00640 1 -- 172.21.15.81:0/3631066128 wait complete. 2024-05-27T06:45:08.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 Processor -- start 2024-05-27T06:45:08.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- start start 2024-05-27T06:45:08.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 0x7f18c8157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 0x7f18c8158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f18c80c7dd0 con 0x7f18c80c5b60 2024-05-27T06:45:08.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f18c80c75c0 con 0x7f18c8072430 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f18c80c5610 con 0x7f18c8161cd0 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57136/0 (socket says 172.21.15.81:57136) 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 -- 172.21.15.81:0/3746055625 learned_addr learned my addr 172.21.15.81:0/3746055625 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18c6a00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 0x7f18c8157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18c7400640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 0x7f18c8158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 -- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 msgr2=0x7f18c8158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 0x7f18c8158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 -- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 msgr2=0x7f18c8157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 0x7f18c8157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 -- 172.21.15.81:0/3746055625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c8157e50 con 0x7f18c80c5b60 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18c7400640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 0x7f18c8158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:08.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18c6a00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 0x7f18c8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bfe00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f18b00190b0 tx=0x7f18b001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b00b0050 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b00bc070 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b00b5830 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c81f9380 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.558+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81f9940 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c806a560 con 0x7f18c80c5b60 2024-05-27T06:45:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f18b00b59d0 con 0x7f18c80c5b60 2024-05-27T06:45:08.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18bf400640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 0x7f18a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f18b0129930 con 0x7f18c80c5b60 2024-05-27T06:45:08.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18c6a00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 0x7f18a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.562+0000 7f18c6a00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 0x7f18a4071910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005890 tx=0x7f18b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.566+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f18b0016090 con 0x7f18c80c5b60 2024-05-27T06:45:08.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.726+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18c81f9110 con 0x7f18a406f400 2024-05-27T06:45:08.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.734+0000 7f18bf400640 1 -- 172.21.15.81:0/3746055625 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251914 (secure 0 0 0) 0x7f18c81f9110 con 0x7f18a406f400 2024-05-27T06:45:08.730 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:08.736 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:08.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 msgr2=0x7f18a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 0x7f18a4071910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005890 tx=0x7f18b8009040 comp rx=0 tx=0).stop 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 msgr2=0x7f18c8151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f18b00190b0 tx=0x7f18b001ace0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 shutdown_connections 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18c8161cd0 0x7f18c8158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18a406f400 0x7f18a4071910 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c80c5b60 0x7f18c8151ea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 --2- 172.21.15.81:0/3746055625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18c8072430 0x7f18c8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 >> 172.21.15.81:0/3746055625 conn(0x7f18c8070700 msgr2=0x7f18c80caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 shutdown_connections 2024-05-27T06:45:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.738+0000 7f18cde00640 1 -- 172.21.15.81:0/3746055625 wait complete. 2024-05-27T06:45:08.752 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-27T06:45:07.491384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217992,"kb_used_data":1824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756728,"statfs":{"total":773094113280,"available":772870889472,"internally_reserved":0,"allocated":1867776,"data_stored":1785576,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001533"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620285+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:03.620285+0000","last_peered":"2024-05-27T06:45:03.620285+0000","last_clean":"2024-05-27T06:45:03.620285+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:03.620285+0000","last_undegraded":"2024-05-27T06:45:03.620285+0000","last_fullsized":"2024-05-27T06:45:03.620285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617672+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:03.617672+0000","last_peered":"2024-05-27T06:45:03.617672+0000","last_clean":"2024-05-27T06:45:03.617672+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:03.617672+0000","last_undegraded":"2024-05-27T06:45:03.617672+0000","last_fullsized":"2024-05-27T06:45:03.617672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542582+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:04.542582+0000","last_peered":"2024-05-27T06:45:04.542582+0000","last_clean":"2024-05-27T06:45:04.542582+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:04.542582+0000","last_undegraded":"2024-05-27T06:45:04.542582+0000","last_fullsized":"2024-05-27T06:45:04.542582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619261+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:03.619261+0000","last_peered":"2024-05-27T06:45:03.619261+0000","last_clean":"2024-05-27T06:45:03.619261+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:03.619261+0000","last_undegraded":"2024-05-27T06:45:03.619261+0000","last_fullsized":"2024-05-27T06:45:03.619261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620460+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:03.620460+0000","last_peered":"2024-05-27T06:45:03.620460+0000","last_clean":"2024-05-27T06:45:03.620460+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:03.620460+0000","last_undegraded":"2024-05-27T06:45:03.620460+0000","last_fullsized":"2024-05-27T06:45:03.620460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619485+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:03.619485+0000","last_peered":"2024-05-27T06:45:03.619485+0000","last_clean":"2024-05-27T06:45:03.619485+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:03.619485+0000","last_undegraded":"2024-05-27T06:45:03.619485+0000","last_fullsized":"2024-05-27T06:45:03.619485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618534+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:03.618534+0000","last_peered":"2024-05-27T06:45:03.618534+0000","last_clean":"2024-05-27T06:45:03.618534+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:03.618534+0000","last_undegraded":"2024-05-27T06:45:03.618534+0000","last_fullsized":"2024-05-27T06:45:03.618534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542255+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:04.542255+0000","last_peered":"2024-05-27T06:45:04.542255+0000","last_clean":"2024-05-27T06:45:04.542255+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:04.542255+0000","last_undegraded":"2024-05-27T06:45:04.542255+0000","last_fullsized":"2024-05-27T06:45:04.542255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620365+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:03.620365+0000","last_peered":"2024-05-27T06:45:03.620365+0000","last_clean":"2024-05-27T06:45:03.620365+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:03.620365+0000","last_undegraded":"2024-05-27T06:45:03.620365+0000","last_fullsized":"2024-05-27T06:45:03.620365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542559+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:04.542559+0000","last_peered":"2024-05-27T06:45:04.542559+0000","last_clean":"2024-05-27T06:45:04.542559+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:04.542559+0000","last_undegraded":"2024-05-27T06:45:04.542559+0000","last_fullsized":"2024-05-27T06:45:04.542559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542270+0000","last_change":"2024-05-27T06:45:02.647403+0000","last_active":"2024-05-27T06:45:04.542270+0000","last_peered":"2024-05-27T06:45:04.542270+0000","last_clean":"2024-05-27T06:45:04.542270+0000","last_became_active":"2024-05-27T06:45:02.647061+0000","last_became_peered":"2024-05-27T06:45:02.647061+0000","last_unstale":"2024-05-27T06:45:04.542270+0000","last_undegraded":"2024-05-27T06:45:04.542270+0000","last_fullsized":"2024-05-27T06:45:04.542270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:10:58.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543063+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:04.543063+0000","last_peered":"2024-05-27T06:45:04.543063+0000","last_clean":"2024-05-27T06:45:04.543063+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:04.543063+0000","last_undegraded":"2024-05-27T06:45:04.543063+0000","last_fullsized":"2024-05-27T06:45:04.543063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542091+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:04.542091+0000","last_peered":"2024-05-27T06:45:04.542091+0000","last_clean":"2024-05-27T06:45:04.542091+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:04.542091+0000","last_undegraded":"2024-05-27T06:45:04.542091+0000","last_fullsized":"2024-05-27T06:45:04.542091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542474+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:04.542474+0000","last_peered":"2024-05-27T06:45:04.542474+0000","last_clean":"2024-05-27T06:45:04.542474+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:04.542474+0000","last_undegraded":"2024-05-27T06:45:04.542474+0000","last_fullsized":"2024-05-27T06:45:04.542474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617212+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:03.617212+0000","last_peered":"2024-05-27T06:45:03.617212+0000","last_clean":"2024-05-27T06:45:03.617212+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:03.617212+0000","last_undegraded":"2024-05-27T06:45:03.617212+0000","last_fullsized":"2024-05-27T06:45:03.617212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619654+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:03.619654+0000","last_peered":"2024-05-27T06:45:03.619654+0000","last_clean":"2024-05-27T06:45:03.619654+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:03.619654+0000","last_undegraded":"2024-05-27T06:45:03.619654+0000","last_fullsized":"2024-05-27T06:45:03.619654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620222+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:03.620222+0000","last_peered":"2024-05-27T06:45:03.620222+0000","last_clean":"2024-05-27T06:45:03.620222+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:03.620222+0000","last_undegraded":"2024-05-27T06:45:03.620222+0000","last_fullsized":"2024-05-27T06:45:03.620222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541770+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:04.541770+0000","last_peered":"2024-05-27T06:45:04.541770+0000","last_clean":"2024-05-27T06:45:04.541770+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:04.541770+0000","last_undegraded":"2024-05-27T06:45:04.541770+0000","last_fullsized":"2024-05-27T06:45:04.541770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620170+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:03.620170+0000","last_peered":"2024-05-27T06:45:03.620170+0000","last_clean":"2024-05-27T06:45:03.620170+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:03.620170+0000","last_undegraded":"2024-05-27T06:45:03.620170+0000","last_fullsized":"2024-05-27T06:45:03.620170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542157+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:04.542157+0000","last_peered":"2024-05-27T06:45:04.542157+0000","last_clean":"2024-05-27T06:45:04.542157+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:04.542157+0000","last_undegraded":"2024-05-27T06:45:04.542157+0000","last_fullsized":"2024-05-27T06:45:04.542157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542660+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:04.542660+0000","last_peered":"2024-05-27T06:45:04.542660+0000","last_clean":"2024-05-27T06:45:04.542660+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:04.542660+0000","last_undegraded":"2024-05-27T06:45:04.542660+0000","last_fullsized":"2024-05-27T06:45:04.542660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619623+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:03.619623+0000","last_peered":"2024-05-27T06:45:03.619623+0000","last_clean":"2024-05-27T06:45:03.619623+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:03.619623+0000","last_undegraded":"2024-05-27T06:45:03.619623+0000","last_fullsized":"2024-05-27T06:45:03.619623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542082+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:04.542082+0000","last_peered":"2024-05-27T06:45:04.542082+0000","last_clean":"2024-05-27T06:45:04.542082+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:04.542082+0000","last_undegraded":"2024-05-27T06:45:04.542082+0000","last_fullsized":"2024-05-27T06:45:04.542082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619147+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:03.619147+0000","last_peered":"2024-05-27T06:45:03.619147+0000","last_clean":"2024-05-27T06:45:03.619147+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:03.619147+0000","last_undegraded":"2024-05-27T06:45:03.619147+0000","last_fullsized":"2024-05-27T06:45:03.619147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543112+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:04.543112+0000","last_peered":"2024-05-27T06:45:04.543112+0000","last_clean":"2024-05-27T06:45:04.543112+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:04.543112+0000","last_undegraded":"2024-05-27T06:45:04.543112+0000","last_fullsized":"2024-05-27T06:45:04.543112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542613+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:04.542613+0000","last_peered":"2024-05-27T06:45:04.542613+0000","last_clean":"2024-05-27T06:45:04.542613+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:04.542613+0000","last_undegraded":"2024-05-27T06:45:04.542613+0000","last_fullsized":"2024-05-27T06:45:04.542613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617693+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:03.617693+0000","last_peered":"2024-05-27T06:45:03.617693+0000","last_clean":"2024-05-27T06:45:03.617693+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:03.617693+0000","last_undegraded":"2024-05-27T06:45:03.617693+0000","last_fullsized":"2024-05-27T06:45:03.617693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541962+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:04.541962+0000","last_peered":"2024-05-27T06:45:04.541962+0000","last_clean":"2024-05-27T06:45:04.541962+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:04.541962+0000","last_undegraded":"2024-05-27T06:45:04.541962+0000","last_fullsized":"2024-05-27T06:45:04.541962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542858+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:04.542858+0000","last_peered":"2024-05-27T06:45:04.542858+0000","last_clean":"2024-05-27T06:45:04.542858+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:04.542858+0000","last_undegraded":"2024-05-27T06:45:04.542858+0000","last_fullsized":"2024-05-27T06:45:04.542858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619714+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:03.619714+0000","last_peered":"2024-05-27T06:45:03.619714+0000","last_clean":"2024-05-27T06:45:03.619714+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:03.619714+0000","last_undegraded":"2024-05-27T06:45:03.619714+0000","last_fullsized":"2024-05-27T06:45:03.619714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542030+0000","last_change":"2024-05-27T06:45:02.627394+0000","last_active":"2024-05-27T06:45:04.542030+0000","last_peered":"2024-05-27T06:45:04.542030+0000","last_clean":"2024-05-27T06:45:04.542030+0000","last_became_active":"2024-05-27T06:45:02.626550+0000","last_became_peered":"2024-05-27T06:45:02.626550+0000","last_unstale":"2024-05-27T06:45:04.542030+0000","last_undegraded":"2024-05-27T06:45:04.542030+0000","last_fullsized":"2024-05-27T06:45:04.542030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:01.383493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541845+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:04.541845+0000","last_peered":"2024-05-27T06:45:04.541845+0000","last_clean":"2024-05-27T06:45:04.541845+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:04.541845+0000","last_undegraded":"2024-05-27T06:45:04.541845+0000","last_fullsized":"2024-05-27T06:45:04.541845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542588+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:04.542588+0000","last_peered":"2024-05-27T06:45:04.542588+0000","last_clean":"2024-05-27T06:45:04.542588+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:04.542588+0000","last_undegraded":"2024-05-27T06:45:04.542588+0000","last_fullsized":"2024-05-27T06:45:04.542588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542731+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:04.542731+0000","last_peered":"2024-05-27T06:45:04.542731+0000","last_clean":"2024-05-27T06:45:04.542731+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:04.542731+0000","last_undegraded":"2024-05-27T06:45:04.542731+0000","last_fullsized":"2024-05-27T06:45:04.542731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543053+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:04.543053+0000","last_peered":"2024-05-27T06:45:04.543053+0000","last_clean":"2024-05-27T06:45:04.543053+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:04.543053+0000","last_undegraded":"2024-05-27T06:45:04.543053+0000","last_fullsized":"2024-05-27T06:45:04.543053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541890+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:04.541890+0000","last_peered":"2024-05-27T06:45:04.541890+0000","last_clean":"2024-05-27T06:45:04.541890+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:04.541890+0000","last_undegraded":"2024-05-27T06:45:04.541890+0000","last_fullsized":"2024-05-27T06:45:04.541890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541609+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:04.541609+0000","last_peered":"2024-05-27T06:45:04.541609+0000","last_clean":"2024-05-27T06:45:04.541609+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:04.541609+0000","last_undegraded":"2024-05-27T06:45:04.541609+0000","last_fullsized":"2024-05-27T06:45:04.541609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542424+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:04.542424+0000","last_peered":"2024-05-27T06:45:04.542424+0000","last_clean":"2024-05-27T06:45:04.542424+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:04.542424+0000","last_undegraded":"2024-05-27T06:45:04.542424+0000","last_fullsized":"2024-05-27T06:45:04.542424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619231+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:03.619231+0000","last_peered":"2024-05-27T06:45:03.619231+0000","last_clean":"2024-05-27T06:45:03.619231+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:03.619231+0000","last_undegraded":"2024-05-27T06:45:03.619231+0000","last_fullsized":"2024-05-27T06:45:03.619231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617259+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:03.617259+0000","last_peered":"2024-05-27T06:45:03.617259+0000","last_clean":"2024-05-27T06:45:03.617259+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:03.617259+0000","last_undegraded":"2024-05-27T06:45:03.617259+0000","last_fullsized":"2024-05-27T06:45:03.617259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542316+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:04.542316+0000","last_peered":"2024-05-27T06:45:04.542316+0000","last_clean":"2024-05-27T06:45:04.542316+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:04.542316+0000","last_undegraded":"2024-05-27T06:45:04.542316+0000","last_fullsized":"2024-05-27T06:45:04.542316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619378+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:03.619378+0000","last_peered":"2024-05-27T06:45:03.619378+0000","last_clean":"2024-05-27T06:45:03.619378+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:03.619378+0000","last_undegraded":"2024-05-27T06:45:03.619378+0000","last_fullsized":"2024-05-27T06:45:03.619378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619917+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:03.619917+0000","last_peered":"2024-05-27T06:45:03.619917+0000","last_clean":"2024-05-27T06:45:03.619917+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:03.619917+0000","last_undegraded":"2024-05-27T06:45:03.619917+0000","last_fullsized":"2024-05-27T06:45:03.619917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543182+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:04.543182+0000","last_peered":"2024-05-27T06:45:04.543182+0000","last_clean":"2024-05-27T06:45:04.543182+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:04.543182+0000","last_undegraded":"2024-05-27T06:45:04.543182+0000","last_fullsized":"2024-05-27T06:45:04.543182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617537+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:03.617537+0000","last_peered":"2024-05-27T06:45:03.617537+0000","last_clean":"2024-05-27T06:45:03.617537+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:03.617537+0000","last_undegraded":"2024-05-27T06:45:03.617537+0000","last_fullsized":"2024-05-27T06:45:03.617537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543150+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:04.543150+0000","last_peered":"2024-05-27T06:45:04.543150+0000","last_clean":"2024-05-27T06:45:04.543150+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:04.543150+0000","last_undegraded":"2024-05-27T06:45:04.543150+0000","last_fullsized":"2024-05-27T06:45:04.543150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542054+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:04.542054+0000","last_peered":"2024-05-27T06:45:04.542054+0000","last_clean":"2024-05-27T06:45:04.542054+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:04.542054+0000","last_undegraded":"2024-05-27T06:45:04.542054+0000","last_fullsized":"2024-05-27T06:45:04.542054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618838+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:03.618838+0000","last_peered":"2024-05-27T06:45:03.618838+0000","last_clean":"2024-05-27T06:45:03.618838+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:03.618838+0000","last_undegraded":"2024-05-27T06:45:03.618838+0000","last_fullsized":"2024-05-27T06:45:03.618838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543124+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:04.543124+0000","last_peered":"2024-05-27T06:45:04.543124+0000","last_clean":"2024-05-27T06:45:04.543124+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:04.543124+0000","last_undegraded":"2024-05-27T06:45:04.543124+0000","last_fullsized":"2024-05-27T06:45:04.543124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542669+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:04.542669+0000","last_peered":"2024-05-27T06:45:04.542669+0000","last_clean":"2024-05-27T06:45:04.542669+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:04.542669+0000","last_undegraded":"2024-05-27T06:45:04.542669+0000","last_fullsized":"2024-05-27T06:45:04.542669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619690+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.619690+0000","last_peered":"2024-05-27T06:45:03.619690+0000","last_clean":"2024-05-27T06:45:03.619690+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:03.619690+0000","last_undegraded":"2024-05-27T06:45:03.619690+0000","last_fullsized":"2024-05-27T06:45:03.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619590+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:03.619590+0000","last_peered":"2024-05-27T06:45:03.619590+0000","last_clean":"2024-05-27T06:45:03.619590+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:03.619590+0000","last_undegraded":"2024-05-27T06:45:03.619590+0000","last_fullsized":"2024-05-27T06:45:03.619590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541708+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:04.541708+0000","last_peered":"2024-05-27T06:45:04.541708+0000","last_clean":"2024-05-27T06:45:04.541708+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:04.541708+0000","last_undegraded":"2024-05-27T06:45:04.541708+0000","last_fullsized":"2024-05-27T06:45:04.541708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541996+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:04.541996+0000","last_peered":"2024-05-27T06:45:04.541996+0000","last_clean":"2024-05-27T06:45:04.541996+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:04.541996+0000","last_undegraded":"2024-05-27T06:45:04.541996+0000","last_fullsized":"2024-05-27T06:45:04.541996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541239+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:04.541239+0000","last_peered":"2024-05-27T06:45:04.541239+0000","last_clean":"2024-05-27T06:45:04.541239+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:04.541239+0000","last_undegraded":"2024-05-27T06:45:04.541239+0000","last_fullsized":"2024-05-27T06:45:04.541239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619502+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:03.619502+0000","last_peered":"2024-05-27T06:45:03.619502+0000","last_clean":"2024-05-27T06:45:03.619502+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:03.619502+0000","last_undegraded":"2024-05-27T06:45:03.619502+0000","last_fullsized":"2024-05-27T06:45:03.619502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543138+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:04.543138+0000","last_peered":"2024-05-27T06:45:04.543138+0000","last_clean":"2024-05-27T06:45:04.543138+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:04.543138+0000","last_undegraded":"2024-05-27T06:45:04.543138+0000","last_fullsized":"2024-05-27T06:45:04.543138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620404+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:03.620404+0000","last_peered":"2024-05-27T06:45:03.620404+0000","last_clean":"2024-05-27T06:45:03.620404+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:03.620404+0000","last_undegraded":"2024-05-27T06:45:03.620404+0000","last_fullsized":"2024-05-27T06:45:03.620404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541809+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:04.541809+0000","last_peered":"2024-05-27T06:45:04.541809+0000","last_clean":"2024-05-27T06:45:04.541809+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:04.541809+0000","last_undegraded":"2024-05-27T06:45:04.541809+0000","last_fullsized":"2024-05-27T06:45:04.541809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620405+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:03.620405+0000","last_peered":"2024-05-27T06:45:03.620405+0000","last_clean":"2024-05-27T06:45:03.620405+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:03.620405+0000","last_undegraded":"2024-05-27T06:45:03.620405+0000","last_fullsized":"2024-05-27T06:45:03.620405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617639+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:03.617639+0000","last_peered":"2024-05-27T06:45:03.617639+0000","last_clean":"2024-05-27T06:45:03.617639+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:03.617639+0000","last_undegraded":"2024-05-27T06:45:03.617639+0000","last_fullsized":"2024-05-27T06:45:03.617639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619736+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:03.619736+0000","last_peered":"2024-05-27T06:45:03.619736+0000","last_clean":"2024-05-27T06:45:03.619736+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:03.619736+0000","last_undegraded":"2024-05-27T06:45:03.619736+0000","last_fullsized":"2024-05-27T06:45:03.619736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541528+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:04.541528+0000","last_peered":"2024-05-27T06:45:04.541528+0000","last_clean":"2024-05-27T06:45:04.541528+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:04.541528+0000","last_undegraded":"2024-05-27T06:45:04.541528+0000","last_fullsized":"2024-05-27T06:45:04.541528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620105+0000","last_change":"2024-05-27T06:45:02.651256+0000","last_active":"2024-05-27T06:45:03.620105+0000","last_peered":"2024-05-27T06:45:03.620105+0000","last_clean":"2024-05-27T06:45:03.620105+0000","last_became_active":"2024-05-27T06:45:02.650646+0000","last_became_peered":"2024-05-27T06:45:02.650646+0000","last_unstale":"2024-05-27T06:45:03.620105+0000","last_undegraded":"2024-05-27T06:45:03.620105+0000","last_fullsized":"2024-05-27T06:45:03.620105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:39:57.469688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618689+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:03.618689+0000","last_peered":"2024-05-27T06:45:03.618689+0000","last_clean":"2024-05-27T06:45:03.618689+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:03.618689+0000","last_undegraded":"2024-05-27T06:45:03.618689+0000","last_fullsized":"2024-05-27T06:45:03.618689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619124+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:03.619124+0000","last_peered":"2024-05-27T06:45:03.619124+0000","last_clean":"2024-05-27T06:45:03.619124+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:03.619124+0000","last_undegraded":"2024-05-27T06:45:03.619124+0000","last_fullsized":"2024-05-27T06:45:03.619124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541730+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:04.541730+0000","last_peered":"2024-05-27T06:45:04.541730+0000","last_clean":"2024-05-27T06:45:04.541730+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:04.541730+0000","last_undegraded":"2024-05-27T06:45:04.541730+0000","last_fullsized":"2024-05-27T06:45:04.541730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618565+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:03.618565+0000","last_peered":"2024-05-27T06:45:03.618565+0000","last_clean":"2024-05-27T06:45:03.618565+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:03.618565+0000","last_undegraded":"2024-05-27T06:45:03.618565+0000","last_fullsized":"2024-05-27T06:45:03.618565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543183+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:04.543183+0000","last_peered":"2024-05-27T06:45:04.543183+0000","last_clean":"2024-05-27T06:45:04.543183+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:04.543183+0000","last_undegraded":"2024-05-27T06:45:04.543183+0000","last_fullsized":"2024-05-27T06:45:04.543183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542094+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:04.542094+0000","last_peered":"2024-05-27T06:45:04.542094+0000","last_clean":"2024-05-27T06:45:04.542094+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:04.542094+0000","last_undegraded":"2024-05-27T06:45:04.542094+0000","last_fullsized":"2024-05-27T06:45:04.542094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617210+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:03.617210+0000","last_peered":"2024-05-27T06:45:03.617210+0000","last_clean":"2024-05-27T06:45:03.617210+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:03.617210+0000","last_undegraded":"2024-05-27T06:45:03.617210+0000","last_fullsized":"2024-05-27T06:45:03.617210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619757+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:03.619757+0000","last_peered":"2024-05-27T06:45:03.619757+0000","last_clean":"2024-05-27T06:45:03.619757+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:03.619757+0000","last_undegraded":"2024-05-27T06:45:03.619757+0000","last_fullsized":"2024-05-27T06:45:03.619757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542806+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:04.542806+0000","last_peered":"2024-05-27T06:45:04.542806+0000","last_clean":"2024-05-27T06:45:04.542806+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:04.542806+0000","last_undegraded":"2024-05-27T06:45:04.542806+0000","last_fullsized":"2024-05-27T06:45:04.542806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617419+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:03.617419+0000","last_peered":"2024-05-27T06:45:03.617419+0000","last_clean":"2024-05-27T06:45:03.617419+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:03.617419+0000","last_undegraded":"2024-05-27T06:45:03.617419+0000","last_fullsized":"2024-05-27T06:45:03.617419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619493+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:03.619493+0000","last_peered":"2024-05-27T06:45:03.619493+0000","last_clean":"2024-05-27T06:45:03.619493+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:03.619493+0000","last_undegraded":"2024-05-27T06:45:03.619493+0000","last_fullsized":"2024-05-27T06:45:03.619493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620147+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.620147+0000","last_peered":"2024-05-27T06:45:03.620147+0000","last_clean":"2024-05-27T06:45:03.620147+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:03.620147+0000","last_undegraded":"2024-05-27T06:45:03.620147+0000","last_fullsized":"2024-05-27T06:45:03.620147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617526+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:03.617526+0000","last_peered":"2024-05-27T06:45:03.617526+0000","last_clean":"2024-05-27T06:45:03.617526+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:03.617526+0000","last_undegraded":"2024-05-27T06:45:03.617526+0000","last_fullsized":"2024-05-27T06:45:03.617526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617705+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:03.617705+0000","last_peered":"2024-05-27T06:45:03.617705+0000","last_clean":"2024-05-27T06:45:03.617705+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:03.617705+0000","last_undegraded":"2024-05-27T06:45:03.617705+0000","last_fullsized":"2024-05-27T06:45:03.617705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543116+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:04.543116+0000","last_peered":"2024-05-27T06:45:04.543116+0000","last_clean":"2024-05-27T06:45:04.543116+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:04.543116+0000","last_undegraded":"2024-05-27T06:45:04.543116+0000","last_fullsized":"2024-05-27T06:45:04.543116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542444+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:04.542444+0000","last_peered":"2024-05-27T06:45:04.542444+0000","last_clean":"2024-05-27T06:45:04.542444+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:04.542444+0000","last_undegraded":"2024-05-27T06:45:04.542444+0000","last_fullsized":"2024-05-27T06:45:04.542444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542819+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:04.542819+0000","last_peered":"2024-05-27T06:45:04.542819+0000","last_clean":"2024-05-27T06:45:04.542819+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:04.542819+0000","last_undegraded":"2024-05-27T06:45:04.542819+0000","last_fullsized":"2024-05-27T06:45:04.542819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618413+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:03.618413+0000","last_peered":"2024-05-27T06:45:03.618413+0000","last_clean":"2024-05-27T06:45:03.618413+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:03.618413+0000","last_undegraded":"2024-05-27T06:45:03.618413+0000","last_fullsized":"2024-05-27T06:45:03.618413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542399+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:04.542399+0000","last_peered":"2024-05-27T06:45:04.542399+0000","last_clean":"2024-05-27T06:45:04.542399+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:04.542399+0000","last_undegraded":"2024-05-27T06:45:04.542399+0000","last_fullsized":"2024-05-27T06:45:04.542399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617281+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:03.617281+0000","last_peered":"2024-05-27T06:45:03.617281+0000","last_clean":"2024-05-27T06:45:03.617281+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:03.617281+0000","last_undegraded":"2024-05-27T06:45:03.617281+0000","last_fullsized":"2024-05-27T06:45:03.617281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542603+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:04.542603+0000","last_peered":"2024-05-27T06:45:04.542603+0000","last_clean":"2024-05-27T06:45:04.542603+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:04.542603+0000","last_undegraded":"2024-05-27T06:45:04.542603+0000","last_fullsized":"2024-05-27T06:45:04.542603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617374+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:03.617374+0000","last_peered":"2024-05-27T06:45:03.617374+0000","last_clean":"2024-05-27T06:45:03.617374+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:03.617374+0000","last_undegraded":"2024-05-27T06:45:03.617374+0000","last_fullsized":"2024-05-27T06:45:03.617374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543008+0000","last_change":"2024-05-27T06:45:02.632527+0000","last_active":"2024-05-27T06:45:04.543008+0000","last_peered":"2024-05-27T06:45:04.543008+0000","last_clean":"2024-05-27T06:45:04.543008+0000","last_became_active":"2024-05-27T06:45:02.632140+0000","last_became_peered":"2024-05-27T06:45:02.632140+0000","last_unstale":"2024-05-27T06:45:04.543008+0000","last_undegraded":"2024-05-27T06:45:04.543008+0000","last_fullsized":"2024-05-27T06:45:04.543008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619660+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:03.619660+0000","last_peered":"2024-05-27T06:45:03.619660+0000","last_clean":"2024-05-27T06:45:03.619660+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:03.619660+0000","last_undegraded":"2024-05-27T06:45:03.619660+0000","last_fullsized":"2024-05-27T06:45:03.619660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541876+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:04.541876+0000","last_peered":"2024-05-27T06:45:04.541876+0000","last_clean":"2024-05-27T06:45:04.541876+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:04.541876+0000","last_undegraded":"2024-05-27T06:45:04.541876+0000","last_fullsized":"2024-05-27T06:45:04.541876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619831+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:03.619831+0000","last_peered":"2024-05-27T06:45:03.619831+0000","last_clean":"2024-05-27T06:45:03.619831+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:03.619831+0000","last_undegraded":"2024-05-27T06:45:03.619831+0000","last_fullsized":"2024-05-27T06:45:03.619831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618795+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:03.618795+0000","last_peered":"2024-05-27T06:45:03.618795+0000","last_clean":"2024-05-27T06:45:03.618795+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:03.618795+0000","last_undegraded":"2024-05-27T06:45:03.618795+0000","last_fullsized":"2024-05-27T06:45:03.618795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619460+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:03.619460+0000","last_peered":"2024-05-27T06:45:03.619460+0000","last_clean":"2024-05-27T06:45:03.619460+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:03.619460+0000","last_undegraded":"2024-05-27T06:45:03.619460+0000","last_fullsized":"2024-05-27T06:45:03.619460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619922+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:03.619922+0000","last_peered":"2024-05-27T06:45:03.619922+0000","last_clean":"2024-05-27T06:45:03.619922+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:03.619922+0000","last_undegraded":"2024-05-27T06:45:03.619922+0000","last_fullsized":"2024-05-27T06:45:03.619922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542992+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:04.542992+0000","last_peered":"2024-05-27T06:45:04.542992+0000","last_clean":"2024-05-27T06:45:04.542992+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:04.542992+0000","last_undegraded":"2024-05-27T06:45:04.542992+0000","last_fullsized":"2024-05-27T06:45:04.542992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542924+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:04.542924+0000","last_peered":"2024-05-27T06:45:04.542924+0000","last_clean":"2024-05-27T06:45:04.542924+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:04.542924+0000","last_undegraded":"2024-05-27T06:45:04.542924+0000","last_fullsized":"2024-05-27T06:45:04.542924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541662+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:04.541662+0000","last_peered":"2024-05-27T06:45:04.541662+0000","last_clean":"2024-05-27T06:45:04.541662+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:04.541662+0000","last_undegraded":"2024-05-27T06:45:04.541662+0000","last_fullsized":"2024-05-27T06:45:04.541662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543246+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:04.543246+0000","last_peered":"2024-05-27T06:45:04.543246+0000","last_clean":"2024-05-27T06:45:04.543246+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:04.543246+0000","last_undegraded":"2024-05-27T06:45:04.543246+0000","last_fullsized":"2024-05-27T06:45:04.543246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":665973,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":665973,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:08.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:08.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.946+0000 7fe523a00640 1 Processor -- start 2024-05-27T06:45:08.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.946+0000 7fe523a00640 1 -- start start 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe523a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe51c06e290 0x7fe51c06e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe523a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe51c15fe80 0x7fe51c160290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe523a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51c0d06f0 con 0x7fe51c0c9770 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe523a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c15d5e0 con 0x7fe51c06e290 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe523a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe51c15c9d0 con 0x7fe51c15fe80 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe51c06e290 0x7fe51c06e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe521000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe51c0c9770 0x7fe51c0cfdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37020/0 (socket says 172.21.15.81:37020) 2024-05-27T06:45:08.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe521000640 1 -- 172.21.15.81:0/510326692 learned_addr learned my addr 172.21.15.81:0/510326692 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe521a00640 1 --2- 172.21.15.81:0/510326692 >> v2:172.21.15.106:3300/0 conn(0x7fe51c15fe80 0x7fe51c160290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe51be00640 1 -- 172.21.15.81:0/510326692 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114609674 0 0) 0x7fe50c001120 con 0x7fe51c0c9770 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe51be00640 1 -- 172.21.15.81:0/510326692 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe500003530 con 0x7fe51c0c9770 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 -- 172.21.15.81:0/510326692 >> v2:172.21.15.106:3300/0 conn(0x7fe51c15fe80 msgr2=0x7fe51c160290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 --2- 172.21.15.81:0/510326692 >> v2:172.21.15.106:3300/0 conn(0x7fe51c15fe80 0x7fe51c160290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 -- 172.21.15.81:0/510326692 >> v1:172.21.15.81:6789/0 conn(0x7fe51c0c9770 legacy=0x7fe51c0cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 -- 172.21.15.81:0/510326692 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c15ccb0 con 0x7fe51c06e290 2024-05-27T06:45:08.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.950+0000 7fe513e00640 1 --2- 172.21.15.81:0/510326692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c06e290 0x7fe51c06e6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe51c073e40 tx=0x7fe50401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faed06bfd2fc0706 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe51be00640 1 -- 172.21.15.81:0/510326692 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5040b0070 con 0x7fe51c06e290 2024-05-27T06:45:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe51be00640 1 -- 172.21.15.81:0/510326692 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5040ab600 con 0x7fe51c06e290 2024-05-27T06:45:08.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe51be00640 1 -- 172.21.15.81:0/510326692 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5040ab9d0 con 0x7fe51c06e290 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- 172.21.15.81:0/510326692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c06e290 msgr2=0x7fe51c06e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 --2- 172.21.15.81:0/510326692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c06e290 0x7fe51c06e6a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe51c073e40 tx=0x7fe50401c3f0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- 172.21.15.81:0/510326692 shutdown_connections 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 --2- 172.21.15.81:0/510326692 >> v2:172.21.15.106:3300/0 conn(0x7fe51c15fe80 0x7fe51c160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 --2- 172.21.15.81:0/510326692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c06e290 0x7fe51c06e6a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- 172.21.15.81:0/510326692 >> 172.21.15.81:0/510326692 conn(0x7fe51c068ab0 msgr2=0x7fe51c068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- 172.21.15.81:0/510326692 shutdown_connections 2024-05-27T06:45:08.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- 172.21.15.81:0/510326692 wait complete. 2024-05-27T06:45:08.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 Processor -- start 2024-05-27T06:45:08.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.954+0000 7fe523a00640 1 -- start start 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe51c06e290 0x7fe51c1f9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe51c0c9770 0x7fe51c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c169910 con 0x7fe51c15fe80 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c0c9d50 con 0x7fe51c0c9770 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c169a90 con 0x7fe51c06e290 2024-05-27T06:45:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57158/0 (socket says 172.21.15.81:57158) 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 -- 172.21.15.81:0/1670270196 learned_addr learned my addr 172.21.15.81:0/1670270196 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe513e00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe51c0c9770 0x7fe51c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521000640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe51c06e290 0x7fe51c1f9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 -- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe51c06e290 msgr2=0x7fe51c1f9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe51c06e290 0x7fe51c1f9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 -- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe51c0c9770 msgr2=0x7fe51c154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe51c0c9770 0x7fe51c154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 -- 172.21.15.81:0/1670270196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c1f9970 con 0x7fe51c15fe80 2024-05-27T06:45:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe521a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe514007d40 tx=0x7fe51400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514014420 con 0x7fe51c15fe80 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe514019070 con 0x7fe51c15fe80 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c0cca20 con 0x7fe51c15fe80 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514014820 con 0x7fe51c15fe80 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c0cefa0 con 0x7fe51c15fe80 2024-05-27T06:45:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.958+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c06a560 con 0x7fe51c15fe80 2024-05-27T06:45:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.962+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe5140149c0 con 0x7fe51c15fe80 2024-05-27T06:45:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.962+0000 7fe519600640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 0x7fe4f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.962+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe51408f940 con 0x7fe51c15fe80 2024-05-27T06:45:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.962+0000 7fe521000640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 0x7fe4f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.962+0000 7fe521000640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 0x7fe4f8071910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005890 tx=0x7fe50c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:08.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:08.966+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe51405c210 con 0x7fe51c15fe80 2024-05-27T06:45:09.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.122+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe51c0ccfc0 con 0x7fe4f806f400 2024-05-27T06:45:09.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.126+0000 7fe519600640 1 -- 172.21.15.81:0/1670270196 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251914 (secure 0 0 0) 0x7fe51c0ccfc0 con 0x7fe4f806f400 2024-05-27T06:45:09.125 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:09.126 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:09.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 msgr2=0x7fe4f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 0x7fe4f8071910 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005890 tx=0x7fe50c009040 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 msgr2=0x7fe51c14c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe514007d40 tx=0x7fe51400e4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 shutdown_connections 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4f806f400 0x7fe4f8071910 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe51c15fe80 0x7fe51c14c460 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe51c0c9770 0x7fe51c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 --2- 172.21.15.81:0/1670270196 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe51c06e290 0x7fe51c1f9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 >> 172.21.15.81:0/1670270196 conn(0x7fe51c068ab0 msgr2=0x7fe51c15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 shutdown_connections 2024-05-27T06:45:09.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.130+0000 7fe523a00640 1 -- 172.21.15.81:0/1670270196 wait complete. 2024-05-27T06:45:09.146 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-27T06:45:07.491384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217992,"kb_used_data":1824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756728,"statfs":{"total":773094113280,"available":772870889472,"internally_reserved":0,"allocated":1867776,"data_stored":1785576,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001533"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620285+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:03.620285+0000","last_peered":"2024-05-27T06:45:03.620285+0000","last_clean":"2024-05-27T06:45:03.620285+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:03.620285+0000","last_undegraded":"2024-05-27T06:45:03.620285+0000","last_fullsized":"2024-05-27T06:45:03.620285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617672+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:03.617672+0000","last_peered":"2024-05-27T06:45:03.617672+0000","last_clean":"2024-05-27T06:45:03.617672+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:03.617672+0000","last_undegraded":"2024-05-27T06:45:03.617672+0000","last_fullsized":"2024-05-27T06:45:03.617672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542582+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:04.542582+0000","last_peered":"2024-05-27T06:45:04.542582+0000","last_clean":"2024-05-27T06:45:04.542582+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:04.542582+0000","last_undegraded":"2024-05-27T06:45:04.542582+0000","last_fullsized":"2024-05-27T06:45:04.542582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619261+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:03.619261+0000","last_peered":"2024-05-27T06:45:03.619261+0000","last_clean":"2024-05-27T06:45:03.619261+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:03.619261+0000","last_undegraded":"2024-05-27T06:45:03.619261+0000","last_fullsized":"2024-05-27T06:45:03.619261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620460+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:03.620460+0000","last_peered":"2024-05-27T06:45:03.620460+0000","last_clean":"2024-05-27T06:45:03.620460+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:03.620460+0000","last_undegraded":"2024-05-27T06:45:03.620460+0000","last_fullsized":"2024-05-27T06:45:03.620460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619485+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:03.619485+0000","last_peered":"2024-05-27T06:45:03.619485+0000","last_clean":"2024-05-27T06:45:03.619485+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:03.619485+0000","last_undegraded":"2024-05-27T06:45:03.619485+0000","last_fullsized":"2024-05-27T06:45:03.619485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618534+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:03.618534+0000","last_peered":"2024-05-27T06:45:03.618534+0000","last_clean":"2024-05-27T06:45:03.618534+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:03.618534+0000","last_undegraded":"2024-05-27T06:45:03.618534+0000","last_fullsized":"2024-05-27T06:45:03.618534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542255+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:04.542255+0000","last_peered":"2024-05-27T06:45:04.542255+0000","last_clean":"2024-05-27T06:45:04.542255+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:04.542255+0000","last_undegraded":"2024-05-27T06:45:04.542255+0000","last_fullsized":"2024-05-27T06:45:04.542255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620365+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:03.620365+0000","last_peered":"2024-05-27T06:45:03.620365+0000","last_clean":"2024-05-27T06:45:03.620365+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:03.620365+0000","last_undegraded":"2024-05-27T06:45:03.620365+0000","last_fullsized":"2024-05-27T06:45:03.620365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542559+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:04.542559+0000","last_peered":"2024-05-27T06:45:04.542559+0000","last_clean":"2024-05-27T06:45:04.542559+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:04.542559+0000","last_undegraded":"2024-05-27T06:45:04.542559+0000","last_fullsized":"2024-05-27T06:45:04.542559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542270+0000","last_change":"2024-05-27T06:45:02.647403+0000","last_active":"2024-05-27T06:45:04.542270+0000","last_peered":"2024-05-27T06:45:04.542270+0000","last_clean":"2024-05-27T06:45:04.542270+0000","last_became_active":"2024-05-27T06:45:02.647061+0000","last_became_peered":"2024-05-27T06:45:02.647061+0000","last_unstale":"2024-05-27T06:45:04.542270+0000","last_undegraded":"2024-05-27T06:45:04.542270+0000","last_fullsized":"2024-05-27T06:45:04.542270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:10:58.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543063+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:04.543063+0000","last_peered":"2024-05-27T06:45:04.543063+0000","last_clean":"2024-05-27T06:45:04.543063+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:04.543063+0000","last_undegraded":"2024-05-27T06:45:04.543063+0000","last_fullsized":"2024-05-27T06:45:04.543063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542091+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:04.542091+0000","last_peered":"2024-05-27T06:45:04.542091+0000","last_clean":"2024-05-27T06:45:04.542091+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:04.542091+0000","last_undegraded":"2024-05-27T06:45:04.542091+0000","last_fullsized":"2024-05-27T06:45:04.542091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542474+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:04.542474+0000","last_peered":"2024-05-27T06:45:04.542474+0000","last_clean":"2024-05-27T06:45:04.542474+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:04.542474+0000","last_undegraded":"2024-05-27T06:45:04.542474+0000","last_fullsized":"2024-05-27T06:45:04.542474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617212+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:03.617212+0000","last_peered":"2024-05-27T06:45:03.617212+0000","last_clean":"2024-05-27T06:45:03.617212+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:03.617212+0000","last_undegraded":"2024-05-27T06:45:03.617212+0000","last_fullsized":"2024-05-27T06:45:03.617212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619654+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:03.619654+0000","last_peered":"2024-05-27T06:45:03.619654+0000","last_clean":"2024-05-27T06:45:03.619654+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:03.619654+0000","last_undegraded":"2024-05-27T06:45:03.619654+0000","last_fullsized":"2024-05-27T06:45:03.619654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620222+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:03.620222+0000","last_peered":"2024-05-27T06:45:03.620222+0000","last_clean":"2024-05-27T06:45:03.620222+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:03.620222+0000","last_undegraded":"2024-05-27T06:45:03.620222+0000","last_fullsized":"2024-05-27T06:45:03.620222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541770+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:04.541770+0000","last_peered":"2024-05-27T06:45:04.541770+0000","last_clean":"2024-05-27T06:45:04.541770+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:04.541770+0000","last_undegraded":"2024-05-27T06:45:04.541770+0000","last_fullsized":"2024-05-27T06:45:04.541770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620170+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:03.620170+0000","last_peered":"2024-05-27T06:45:03.620170+0000","last_clean":"2024-05-27T06:45:03.620170+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:03.620170+0000","last_undegraded":"2024-05-27T06:45:03.620170+0000","last_fullsized":"2024-05-27T06:45:03.620170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542157+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:04.542157+0000","last_peered":"2024-05-27T06:45:04.542157+0000","last_clean":"2024-05-27T06:45:04.542157+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:04.542157+0000","last_undegraded":"2024-05-27T06:45:04.542157+0000","last_fullsized":"2024-05-27T06:45:04.542157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542660+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:04.542660+0000","last_peered":"2024-05-27T06:45:04.542660+0000","last_clean":"2024-05-27T06:45:04.542660+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:04.542660+0000","last_undegraded":"2024-05-27T06:45:04.542660+0000","last_fullsized":"2024-05-27T06:45:04.542660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619623+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:03.619623+0000","last_peered":"2024-05-27T06:45:03.619623+0000","last_clean":"2024-05-27T06:45:03.619623+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:03.619623+0000","last_undegraded":"2024-05-27T06:45:03.619623+0000","last_fullsized":"2024-05-27T06:45:03.619623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542082+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:04.542082+0000","last_peered":"2024-05-27T06:45:04.542082+0000","last_clean":"2024-05-27T06:45:04.542082+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:04.542082+0000","last_undegraded":"2024-05-27T06:45:04.542082+0000","last_fullsized":"2024-05-27T06:45:04.542082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619147+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:03.619147+0000","last_peered":"2024-05-27T06:45:03.619147+0000","last_clean":"2024-05-27T06:45:03.619147+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:03.619147+0000","last_undegraded":"2024-05-27T06:45:03.619147+0000","last_fullsized":"2024-05-27T06:45:03.619147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543112+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:04.543112+0000","last_peered":"2024-05-27T06:45:04.543112+0000","last_clean":"2024-05-27T06:45:04.543112+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:04.543112+0000","last_undegraded":"2024-05-27T06:45:04.543112+0000","last_fullsized":"2024-05-27T06:45:04.543112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542613+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:04.542613+0000","last_peered":"2024-05-27T06:45:04.542613+0000","last_clean":"2024-05-27T06:45:04.542613+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:04.542613+0000","last_undegraded":"2024-05-27T06:45:04.542613+0000","last_fullsized":"2024-05-27T06:45:04.542613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617693+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:03.617693+0000","last_peered":"2024-05-27T06:45:03.617693+0000","last_clean":"2024-05-27T06:45:03.617693+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:03.617693+0000","last_undegraded":"2024-05-27T06:45:03.617693+0000","last_fullsized":"2024-05-27T06:45:03.617693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541962+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:04.541962+0000","last_peered":"2024-05-27T06:45:04.541962+0000","last_clean":"2024-05-27T06:45:04.541962+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:04.541962+0000","last_undegraded":"2024-05-27T06:45:04.541962+0000","last_fullsized":"2024-05-27T06:45:04.541962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542858+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:04.542858+0000","last_peered":"2024-05-27T06:45:04.542858+0000","last_clean":"2024-05-27T06:45:04.542858+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:04.542858+0000","last_undegraded":"2024-05-27T06:45:04.542858+0000","last_fullsized":"2024-05-27T06:45:04.542858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619714+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:03.619714+0000","last_peered":"2024-05-27T06:45:03.619714+0000","last_clean":"2024-05-27T06:45:03.619714+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:03.619714+0000","last_undegraded":"2024-05-27T06:45:03.619714+0000","last_fullsized":"2024-05-27T06:45:03.619714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542030+0000","last_change":"2024-05-27T06:45:02.627394+0000","last_active":"2024-05-27T06:45:04.542030+0000","last_peered":"2024-05-27T06:45:04.542030+0000","last_clean":"2024-05-27T06:45:04.542030+0000","last_became_active":"2024-05-27T06:45:02.626550+0000","last_became_peered":"2024-05-27T06:45:02.626550+0000","last_unstale":"2024-05-27T06:45:04.542030+0000","last_undegraded":"2024-05-27T06:45:04.542030+0000","last_fullsized":"2024-05-27T06:45:04.542030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:01.383493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541845+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:04.541845+0000","last_peered":"2024-05-27T06:45:04.541845+0000","last_clean":"2024-05-27T06:45:04.541845+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:04.541845+0000","last_undegraded":"2024-05-27T06:45:04.541845+0000","last_fullsized":"2024-05-27T06:45:04.541845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542588+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:04.542588+0000","last_peered":"2024-05-27T06:45:04.542588+0000","last_clean":"2024-05-27T06:45:04.542588+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:04.542588+0000","last_undegraded":"2024-05-27T06:45:04.542588+0000","last_fullsized":"2024-05-27T06:45:04.542588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542731+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:04.542731+0000","last_peered":"2024-05-27T06:45:04.542731+0000","last_clean":"2024-05-27T06:45:04.542731+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:04.542731+0000","last_undegraded":"2024-05-27T06:45:04.542731+0000","last_fullsized":"2024-05-27T06:45:04.542731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543053+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:04.543053+0000","last_peered":"2024-05-27T06:45:04.543053+0000","last_clean":"2024-05-27T06:45:04.543053+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:04.543053+0000","last_undegraded":"2024-05-27T06:45:04.543053+0000","last_fullsized":"2024-05-27T06:45:04.543053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541890+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:04.541890+0000","last_peered":"2024-05-27T06:45:04.541890+0000","last_clean":"2024-05-27T06:45:04.541890+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:04.541890+0000","last_undegraded":"2024-05-27T06:45:04.541890+0000","last_fullsized":"2024-05-27T06:45:04.541890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541609+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:04.541609+0000","last_peered":"2024-05-27T06:45:04.541609+0000","last_clean":"2024-05-27T06:45:04.541609+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:04.541609+0000","last_undegraded":"2024-05-27T06:45:04.541609+0000","last_fullsized":"2024-05-27T06:45:04.541609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542424+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:04.542424+0000","last_peered":"2024-05-27T06:45:04.542424+0000","last_clean":"2024-05-27T06:45:04.542424+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:04.542424+0000","last_undegraded":"2024-05-27T06:45:04.542424+0000","last_fullsized":"2024-05-27T06:45:04.542424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619231+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:03.619231+0000","last_peered":"2024-05-27T06:45:03.619231+0000","last_clean":"2024-05-27T06:45:03.619231+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:03.619231+0000","last_undegraded":"2024-05-27T06:45:03.619231+0000","last_fullsized":"2024-05-27T06:45:03.619231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617259+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:03.617259+0000","last_peered":"2024-05-27T06:45:03.617259+0000","last_clean":"2024-05-27T06:45:03.617259+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:03.617259+0000","last_undegraded":"2024-05-27T06:45:03.617259+0000","last_fullsized":"2024-05-27T06:45:03.617259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542316+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:04.542316+0000","last_peered":"2024-05-27T06:45:04.542316+0000","last_clean":"2024-05-27T06:45:04.542316+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:04.542316+0000","last_undegraded":"2024-05-27T06:45:04.542316+0000","last_fullsized":"2024-05-27T06:45:04.542316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619378+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:03.619378+0000","last_peered":"2024-05-27T06:45:03.619378+0000","last_clean":"2024-05-27T06:45:03.619378+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:03.619378+0000","last_undegraded":"2024-05-27T06:45:03.619378+0000","last_fullsized":"2024-05-27T06:45:03.619378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619917+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:03.619917+0000","last_peered":"2024-05-27T06:45:03.619917+0000","last_clean":"2024-05-27T06:45:03.619917+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:03.619917+0000","last_undegraded":"2024-05-27T06:45:03.619917+0000","last_fullsized":"2024-05-27T06:45:03.619917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543182+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:04.543182+0000","last_peered":"2024-05-27T06:45:04.543182+0000","last_clean":"2024-05-27T06:45:04.543182+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:04.543182+0000","last_undegraded":"2024-05-27T06:45:04.543182+0000","last_fullsized":"2024-05-27T06:45:04.543182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617537+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:03.617537+0000","last_peered":"2024-05-27T06:45:03.617537+0000","last_clean":"2024-05-27T06:45:03.617537+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:03.617537+0000","last_undegraded":"2024-05-27T06:45:03.617537+0000","last_fullsized":"2024-05-27T06:45:03.617537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543150+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:04.543150+0000","last_peered":"2024-05-27T06:45:04.543150+0000","last_clean":"2024-05-27T06:45:04.543150+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:04.543150+0000","last_undegraded":"2024-05-27T06:45:04.543150+0000","last_fullsized":"2024-05-27T06:45:04.543150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542054+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:04.542054+0000","last_peered":"2024-05-27T06:45:04.542054+0000","last_clean":"2024-05-27T06:45:04.542054+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:04.542054+0000","last_undegraded":"2024-05-27T06:45:04.542054+0000","last_fullsized":"2024-05-27T06:45:04.542054+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618838+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:03.618838+0000","last_peered":"2024-05-27T06:45:03.618838+0000","last_clean":"2024-05-27T06:45:03.618838+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:03.618838+0000","last_undegraded":"2024-05-27T06:45:03.618838+0000","last_fullsized":"2024-05-27T06:45:03.618838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543124+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:04.543124+0000","last_peered":"2024-05-27T06:45:04.543124+0000","last_clean":"2024-05-27T06:45:04.543124+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:04.543124+0000","last_undegraded":"2024-05-27T06:45:04.543124+0000","last_fullsized":"2024-05-27T06:45:04.543124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542669+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:04.542669+0000","last_peered":"2024-05-27T06:45:04.542669+0000","last_clean":"2024-05-27T06:45:04.542669+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:04.542669+0000","last_undegraded":"2024-05-27T06:45:04.542669+0000","last_fullsized":"2024-05-27T06:45:04.542669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619690+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.619690+0000","last_peered":"2024-05-27T06:45:03.619690+0000","last_clean":"2024-05-27T06:45:03.619690+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:03.619690+0000","last_undegraded":"2024-05-27T06:45:03.619690+0000","last_fullsized":"2024-05-27T06:45:03.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619590+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:03.619590+0000","last_peered":"2024-05-27T06:45:03.619590+0000","last_clean":"2024-05-27T06:45:03.619590+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:03.619590+0000","last_undegraded":"2024-05-27T06:45:03.619590+0000","last_fullsized":"2024-05-27T06:45:03.619590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541708+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:04.541708+0000","last_peered":"2024-05-27T06:45:04.541708+0000","last_clean":"2024-05-27T06:45:04.541708+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:04.541708+0000","last_undegraded":"2024-05-27T06:45:04.541708+0000","last_fullsized":"2024-05-27T06:45:04.541708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541996+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:04.541996+0000","last_peered":"2024-05-27T06:45:04.541996+0000","last_clean":"2024-05-27T06:45:04.541996+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:04.541996+0000","last_undegraded":"2024-05-27T06:45:04.541996+0000","last_fullsized":"2024-05-27T06:45:04.541996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541239+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:04.541239+0000","last_peered":"2024-05-27T06:45:04.541239+0000","last_clean":"2024-05-27T06:45:04.541239+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:04.541239+0000","last_undegraded":"2024-05-27T06:45:04.541239+0000","last_fullsized":"2024-05-27T06:45:04.541239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619502+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:03.619502+0000","last_peered":"2024-05-27T06:45:03.619502+0000","last_clean":"2024-05-27T06:45:03.619502+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:03.619502+0000","last_undegraded":"2024-05-27T06:45:03.619502+0000","last_fullsized":"2024-05-27T06:45:03.619502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543138+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:04.543138+0000","last_peered":"2024-05-27T06:45:04.543138+0000","last_clean":"2024-05-27T06:45:04.543138+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:04.543138+0000","last_undegraded":"2024-05-27T06:45:04.543138+0000","last_fullsized":"2024-05-27T06:45:04.543138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620404+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:03.620404+0000","last_peered":"2024-05-27T06:45:03.620404+0000","last_clean":"2024-05-27T06:45:03.620404+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:03.620404+0000","last_undegraded":"2024-05-27T06:45:03.620404+0000","last_fullsized":"2024-05-27T06:45:03.620404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541809+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:04.541809+0000","last_peered":"2024-05-27T06:45:04.541809+0000","last_clean":"2024-05-27T06:45:04.541809+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:04.541809+0000","last_undegraded":"2024-05-27T06:45:04.541809+0000","last_fullsized":"2024-05-27T06:45:04.541809+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620405+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:03.620405+0000","last_peered":"2024-05-27T06:45:03.620405+0000","last_clean":"2024-05-27T06:45:03.620405+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:03.620405+0000","last_undegraded":"2024-05-27T06:45:03.620405+0000","last_fullsized":"2024-05-27T06:45:03.620405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617639+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:03.617639+0000","last_peered":"2024-05-27T06:45:03.617639+0000","last_clean":"2024-05-27T06:45:03.617639+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:03.617639+0000","last_undegraded":"2024-05-27T06:45:03.617639+0000","last_fullsized":"2024-05-27T06:45:03.617639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619736+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:03.619736+0000","last_peered":"2024-05-27T06:45:03.619736+0000","last_clean":"2024-05-27T06:45:03.619736+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:03.619736+0000","last_undegraded":"2024-05-27T06:45:03.619736+0000","last_fullsized":"2024-05-27T06:45:03.619736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541528+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:04.541528+0000","last_peered":"2024-05-27T06:45:04.541528+0000","last_clean":"2024-05-27T06:45:04.541528+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:04.541528+0000","last_undegraded":"2024-05-27T06:45:04.541528+0000","last_fullsized":"2024-05-27T06:45:04.541528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620105+0000","last_change":"2024-05-27T06:45:02.651256+0000","last_active":"2024-05-27T06:45:03.620105+0000","last_peered":"2024-05-27T06:45:03.620105+0000","last_clean":"2024-05-27T06:45:03.620105+0000","last_became_active":"2024-05-27T06:45:02.650646+0000","last_became_peered":"2024-05-27T06:45:02.650646+0000","last_unstale":"2024-05-27T06:45:03.620105+0000","last_undegraded":"2024-05-27T06:45:03.620105+0000","last_fullsized":"2024-05-27T06:45:03.620105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:39:57.469688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618689+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:03.618689+0000","last_peered":"2024-05-27T06:45:03.618689+0000","last_clean":"2024-05-27T06:45:03.618689+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:03.618689+0000","last_undegraded":"2024-05-27T06:45:03.618689+0000","last_fullsized":"2024-05-27T06:45:03.618689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619124+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:03.619124+0000","last_peered":"2024-05-27T06:45:03.619124+0000","last_clean":"2024-05-27T06:45:03.619124+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:03.619124+0000","last_undegraded":"2024-05-27T06:45:03.619124+0000","last_fullsized":"2024-05-27T06:45:03.619124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541730+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:04.541730+0000","last_peered":"2024-05-27T06:45:04.541730+0000","last_clean":"2024-05-27T06:45:04.541730+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:04.541730+0000","last_undegraded":"2024-05-27T06:45:04.541730+0000","last_fullsized":"2024-05-27T06:45:04.541730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618565+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:03.618565+0000","last_peered":"2024-05-27T06:45:03.618565+0000","last_clean":"2024-05-27T06:45:03.618565+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:03.618565+0000","last_undegraded":"2024-05-27T06:45:03.618565+0000","last_fullsized":"2024-05-27T06:45:03.618565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543183+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:04.543183+0000","last_peered":"2024-05-27T06:45:04.543183+0000","last_clean":"2024-05-27T06:45:04.543183+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:04.543183+0000","last_undegraded":"2024-05-27T06:45:04.543183+0000","last_fullsized":"2024-05-27T06:45:04.543183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542094+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:04.542094+0000","last_peered":"2024-05-27T06:45:04.542094+0000","last_clean":"2024-05-27T06:45:04.542094+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:04.542094+0000","last_undegraded":"2024-05-27T06:45:04.542094+0000","last_fullsized":"2024-05-27T06:45:04.542094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617210+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:03.617210+0000","last_peered":"2024-05-27T06:45:03.617210+0000","last_clean":"2024-05-27T06:45:03.617210+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:03.617210+0000","last_undegraded":"2024-05-27T06:45:03.617210+0000","last_fullsized":"2024-05-27T06:45:03.617210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619757+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:03.619757+0000","last_peered":"2024-05-27T06:45:03.619757+0000","last_clean":"2024-05-27T06:45:03.619757+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:03.619757+0000","last_undegraded":"2024-05-27T06:45:03.619757+0000","last_fullsized":"2024-05-27T06:45:03.619757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542806+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:04.542806+0000","last_peered":"2024-05-27T06:45:04.542806+0000","last_clean":"2024-05-27T06:45:04.542806+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:04.542806+0000","last_undegraded":"2024-05-27T06:45:04.542806+0000","last_fullsized":"2024-05-27T06:45:04.542806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617419+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:03.617419+0000","last_peered":"2024-05-27T06:45:03.617419+0000","last_clean":"2024-05-27T06:45:03.617419+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:03.617419+0000","last_undegraded":"2024-05-27T06:45:03.617419+0000","last_fullsized":"2024-05-27T06:45:03.617419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619493+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:03.619493+0000","last_peered":"2024-05-27T06:45:03.619493+0000","last_clean":"2024-05-27T06:45:03.619493+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:03.619493+0000","last_undegraded":"2024-05-27T06:45:03.619493+0000","last_fullsized":"2024-05-27T06:45:03.619493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620147+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.620147+0000","last_peered":"2024-05-27T06:45:03.620147+0000","last_clean":"2024-05-27T06:45:03.620147+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:03.620147+0000","last_undegraded":"2024-05-27T06:45:03.620147+0000","last_fullsized":"2024-05-27T06:45:03.620147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617526+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:03.617526+0000","last_peered":"2024-05-27T06:45:03.617526+0000","last_clean":"2024-05-27T06:45:03.617526+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:03.617526+0000","last_undegraded":"2024-05-27T06:45:03.617526+0000","last_fullsized":"2024-05-27T06:45:03.617526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617705+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:03.617705+0000","last_peered":"2024-05-27T06:45:03.617705+0000","last_clean":"2024-05-27T06:45:03.617705+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:03.617705+0000","last_undegraded":"2024-05-27T06:45:03.617705+0000","last_fullsized":"2024-05-27T06:45:03.617705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543116+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:04.543116+0000","last_peered":"2024-05-27T06:45:04.543116+0000","last_clean":"2024-05-27T06:45:04.543116+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:04.543116+0000","last_undegraded":"2024-05-27T06:45:04.543116+0000","last_fullsized":"2024-05-27T06:45:04.543116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542444+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:04.542444+0000","last_peered":"2024-05-27T06:45:04.542444+0000","last_clean":"2024-05-27T06:45:04.542444+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:04.542444+0000","last_undegraded":"2024-05-27T06:45:04.542444+0000","last_fullsized":"2024-05-27T06:45:04.542444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542819+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:04.542819+0000","last_peered":"2024-05-27T06:45:04.542819+0000","last_clean":"2024-05-27T06:45:04.542819+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:04.542819+0000","last_undegraded":"2024-05-27T06:45:04.542819+0000","last_fullsized":"2024-05-27T06:45:04.542819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618413+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:03.618413+0000","last_peered":"2024-05-27T06:45:03.618413+0000","last_clean":"2024-05-27T06:45:03.618413+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:03.618413+0000","last_undegraded":"2024-05-27T06:45:03.618413+0000","last_fullsized":"2024-05-27T06:45:03.618413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542399+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:04.542399+0000","last_peered":"2024-05-27T06:45:04.542399+0000","last_clean":"2024-05-27T06:45:04.542399+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:04.542399+0000","last_undegraded":"2024-05-27T06:45:04.542399+0000","last_fullsized":"2024-05-27T06:45:04.542399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617281+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:03.617281+0000","last_peered":"2024-05-27T06:45:03.617281+0000","last_clean":"2024-05-27T06:45:03.617281+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:03.617281+0000","last_undegraded":"2024-05-27T06:45:03.617281+0000","last_fullsized":"2024-05-27T06:45:03.617281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542603+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:04.542603+0000","last_peered":"2024-05-27T06:45:04.542603+0000","last_clean":"2024-05-27T06:45:04.542603+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:04.542603+0000","last_undegraded":"2024-05-27T06:45:04.542603+0000","last_fullsized":"2024-05-27T06:45:04.542603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617374+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:03.617374+0000","last_peered":"2024-05-27T06:45:03.617374+0000","last_clean":"2024-05-27T06:45:03.617374+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:03.617374+0000","last_undegraded":"2024-05-27T06:45:03.617374+0000","last_fullsized":"2024-05-27T06:45:03.617374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543008+0000","last_change":"2024-05-27T06:45:02.632527+0000","last_active":"2024-05-27T06:45:04.543008+0000","last_peered":"2024-05-27T06:45:04.543008+0000","last_clean":"2024-05-27T06:45:04.543008+0000","last_became_active":"2024-05-27T06:45:02.632140+0000","last_became_peered":"2024-05-27T06:45:02.632140+0000","last_unstale":"2024-05-27T06:45:04.543008+0000","last_undegraded":"2024-05-27T06:45:04.543008+0000","last_fullsized":"2024-05-27T06:45:04.543008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619660+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:03.619660+0000","last_peered":"2024-05-27T06:45:03.619660+0000","last_clean":"2024-05-27T06:45:03.619660+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:03.619660+0000","last_undegraded":"2024-05-27T06:45:03.619660+0000","last_fullsized":"2024-05-27T06:45:03.619660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541876+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:04.541876+0000","last_peered":"2024-05-27T06:45:04.541876+0000","last_clean":"2024-05-27T06:45:04.541876+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:04.541876+0000","last_undegraded":"2024-05-27T06:45:04.541876+0000","last_fullsized":"2024-05-27T06:45:04.541876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619831+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:03.619831+0000","last_peered":"2024-05-27T06:45:03.619831+0000","last_clean":"2024-05-27T06:45:03.619831+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:03.619831+0000","last_undegraded":"2024-05-27T06:45:03.619831+0000","last_fullsized":"2024-05-27T06:45:03.619831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618795+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:03.618795+0000","last_peered":"2024-05-27T06:45:03.618795+0000","last_clean":"2024-05-27T06:45:03.618795+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:03.618795+0000","last_undegraded":"2024-05-27T06:45:03.618795+0000","last_fullsized":"2024-05-27T06:45:03.618795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619460+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:03.619460+0000","last_peered":"2024-05-27T06:45:03.619460+0000","last_clean":"2024-05-27T06:45:03.619460+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:03.619460+0000","last_undegraded":"2024-05-27T06:45:03.619460+0000","last_fullsized":"2024-05-27T06:45:03.619460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619922+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:03.619922+0000","last_peered":"2024-05-27T06:45:03.619922+0000","last_clean":"2024-05-27T06:45:03.619922+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:03.619922+0000","last_undegraded":"2024-05-27T06:45:03.619922+0000","last_fullsized":"2024-05-27T06:45:03.619922+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542992+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:04.542992+0000","last_peered":"2024-05-27T06:45:04.542992+0000","last_clean":"2024-05-27T06:45:04.542992+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:04.542992+0000","last_undegraded":"2024-05-27T06:45:04.542992+0000","last_fullsized":"2024-05-27T06:45:04.542992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542924+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:04.542924+0000","last_peered":"2024-05-27T06:45:04.542924+0000","last_clean":"2024-05-27T06:45:04.542924+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:04.542924+0000","last_undegraded":"2024-05-27T06:45:04.542924+0000","last_fullsized":"2024-05-27T06:45:04.542924+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541662+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:04.541662+0000","last_peered":"2024-05-27T06:45:04.541662+0000","last_clean":"2024-05-27T06:45:04.541662+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:04.541662+0000","last_undegraded":"2024-05-27T06:45:04.541662+0000","last_fullsized":"2024-05-27T06:45:04.541662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543246+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:04.543246+0000","last_peered":"2024-05-27T06:45:04.543246+0000","last_clean":"2024-05-27T06:45:04.543246+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:04.543246+0000","last_undegraded":"2024-05-27T06:45:04.543246+0000","last_fullsized":"2024-05-27T06:45:04.543246+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":665973,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":665973,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:09.150 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-27T06:45:09.150 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-27T06:45:09.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-27T06:45:09.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-27T06:45:09.150 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-27T06:45:09.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.342+0000 7f6af8c00640 1 Processor -- start 2024-05-27T06:45:09.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.342+0000 7f6af8c00640 1 -- start start 2024-05-27T06:45:09.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af415fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6af4156930 0x7f6af415cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6af40cfd50 0x7f6af40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6af415cb50 con 0x7f6af4156930 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6af415c850 con 0x7f6af40cfd50 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af8c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6af4160350 con 0x7f6af40c9650 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6af4156930 0x7f6af415cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6af4156930 0x7f6af415cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57170/0 (socket says 172.21.15.81:57170) 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 -- 172.21.15.81:0/1589771011 learned_addr learned my addr 172.21.15.81:0/1589771011 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af1e00640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af415fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6af2800640 1 --2- 172.21.15.81:0/1589771011 >> v2:172.21.15.106:3300/0 conn(0x7f6af40cfd50 0x7f6af40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 -- 172.21.15.81:0/1589771011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 msgr2=0x7f6af415fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af415fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 -- 172.21.15.81:0/1589771011 >> v2:172.21.15.106:3300/0 conn(0x7f6af40cfd50 msgr2=0x7f6af40d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:09.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 --2- 172.21.15.81:0/1589771011 >> v2:172.21.15.106:3300/0 conn(0x7f6af40cfd50 0x7f6af40d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 -- 172.21.15.81:0/1589771011 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af4160610 con 0x7f6af4156930 2024-05-27T06:45:09.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.346+0000 7f6ae9400640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af4156930 0x7f6af415cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6af415a330 tx=0x7f6adc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d205e8e1bbfb73a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af1400640 1 -- 172.21.15.81:0/1589771011 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0b0070 con 0x7f6af4156930 2024-05-27T06:45:09.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af1400640 1 -- 172.21.15.81:0/1589771011 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc0ab600 con 0x7f6af4156930 2024-05-27T06:45:09.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af1400640 1 -- 172.21.15.81:0/1589771011 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0ab9d0 con 0x7f6af4156930 2024-05-27T06:45:09.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- 172.21.15.81:0/1589771011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af4156930 msgr2=0x7f6af415cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af4156930 0x7f6af415cfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6af415a330 tx=0x7f6adc01c520 comp rx=0 tx=0).stop 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- 172.21.15.81:0/1589771011 shutdown_connections 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 --2- 172.21.15.81:0/1589771011 >> v2:172.21.15.106:3300/0 conn(0x7f6af40cfd50 0x7f6af40d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af4156930 0x7f6af415cfe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 --2- 172.21.15.81:0/1589771011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af415fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- 172.21.15.81:0/1589771011 >> 172.21.15.81:0/1589771011 conn(0x7f6af4068ab0 msgr2=0x7f6af4068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- 172.21.15.81:0/1589771011 shutdown_connections 2024-05-27T06:45:09.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- 172.21.15.81:0/1589771011 wait complete. 2024-05-27T06:45:09.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 Processor -- start 2024-05-27T06:45:09.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.350+0000 7f6af8c00640 1 -- start start 2024-05-27T06:45:09.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af41f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 0x7f6af4146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6af4156d40 con 0x7f6af40cfd50 2024-05-27T06:45:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6af415d520 con 0x7f6af4156930 2024-05-27T06:45:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6af41f9a10 con 0x7f6af40c9650 2024-05-27T06:45:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57172/0 (socket says 172.21.15.81:57172) 2024-05-27T06:45:09.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 -- 172.21.15.81:0/2255114626 learned_addr learned my addr 172.21.15.81:0/2255114626 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 -- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 msgr2=0x7f6af41f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af2800640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 0x7f6af4146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af1e00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af41f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af41f9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 -- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 msgr2=0x7f6af4146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 0x7f6af4146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 -- 172.21.15.81:0/2255114626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af40cb920 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af2800640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 0x7f6af4146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af1e00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af41f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6ae9400640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c10 tx=0x7f6adc0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0ba070 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6adc0b43f0 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af41f9540 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0b0040 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af415a4e0 con 0x7f6af40cfd50 2024-05-27T06:45:09.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.354+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af406a560 con 0x7f6af40cfd50 2024-05-27T06:45:09.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.358+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6adc0bd070 con 0x7f6af40cfd50 2024-05-27T06:45:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.358+0000 7f6aeaa00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 0x7f6ad0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.358+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6adc1284c0 con 0x7f6af40cfd50 2024-05-27T06:45:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.358+0000 7f6af1e00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 0x7f6ad0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.358+0000 7f6af1e00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 0x7f6ad0071940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40097a0 tx=0x7f6ae4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.362+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6adc0eff50 con 0x7f6af40cfd50 2024-05-27T06:45:09.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.558+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f6af40cb150 con 0x7f6af40cfd50 2024-05-27T06:45:09.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.562+0000 7f6aeaa00640 1 -- 172.21.15.81:0/2255114626 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+179543 (secure 0 0 0) 0x7f6adc0efd70 con 0x7f6af40cfd50 2024-05-27T06:45:09.558 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 msgr2=0x7f6ad0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 0x7f6ad0071940 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40097a0 tx=0x7f6ae4008040 comp rx=0 tx=0).stop 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 msgr2=0x7f6af4148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c10 tx=0x7f6adc0192c0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 shutdown_connections 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ad006f430 0x7f6ad0071940 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6af4156930 0x7f6af4146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af40cfd50 0x7f6af4148170 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 --2- 172.21.15.81:0/2255114626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6af40c9650 0x7f6af41f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 >> 172.21.15.81:0/2255114626 conn(0x7f6af4068ab0 msgr2=0x7f6af414e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:09.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 shutdown_connections 2024-05-27T06:45:09.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.566+0000 7f6af8c00640 1 -- 172.21.15.81:0/2255114626 wait complete. 2024-05-27T06:45:09.580 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1352206918},{"type":"v1","addr":"172.21.15.81:6833","nonce":1352206918}]},"active_addr":"172.21.15.81:6833/1352206918","active_change":"2024-05-27T06:44:43.476818+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":3989426965}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2840912402}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":941857140}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":684564554}]}]} 2024-05-27T06:45:09.583 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-27T06:45:09.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-27T06:45:09.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:09.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.738+0000 7fbb25200640 1 Processor -- start 2024-05-27T06:45:09.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.738+0000 7fbb25200640 1 -- start start 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb25200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbb20072430 0x7fbb20159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb25200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb20072840 con 0x7fbb20161cd0 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb25200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb200c5270 con 0x7fbb2015a040 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb25200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fbb200c5040 con 0x7fbb20072430 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1e800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fbb20161cd0 0x7fbb201640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37030/0 (socket says 172.21.15.81:37030) 2024-05-27T06:45:09.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1e800640 1 -- 172.21.15.81:0/1572837138 learned_addr learned my addr 172.21.15.81:0/1572837138 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 --2- 172.21.15.81:0/1572837138 >> v2:172.21.15.106:3300/0 conn(0x7fbb20072430 0x7fbb20159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4187295071 0 0) 0x7fbb20072840 con 0x7fbb20161cd0 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb04003530 con 0x7fbb20161cd0 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827150929 0 0) 0x7fbb200c5270 con 0x7fbb2015a040 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb20072840 con 0x7fbb2015a040 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2802952862 0 0) 0x7fbb04003530 con 0x7fbb20161cd0 2024-05-27T06:45:09.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 >> v2:172.21.15.106:3300/0 conn(0x7fbb20072430 msgr2=0x7fbb20159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 --2- 172.21.15.81:0/1572837138 >> v2:172.21.15.106:3300/0 conn(0x7fbb20072430 0x7fbb20159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 >> v1:172.21.15.106:6789/0 conn(0x7fbb2015a040 legacy=0x7fbb200c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb200c7810 con 0x7fbb20161cd0 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fbb18001a80 con 0x7fbb20161cd0 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 >> v1:172.21.15.81:6789/0 conn(0x7fbb20161cd0 legacy=0x7fbb201640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 0x7fbb04004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04004810 0x7fbb04004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb04005290 0x7fbb0400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0400e420 con 0x7fbb04003c40 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0400e5d0 con 0x7fbb04005290 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0400e7d0 con 0x7fbb04004810 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 0x7fbb04004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1e800640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb04005290 0x7fbb0400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb17e00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04004810 0x7fbb04004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 -- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04004810 msgr2=0x7fbb04004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04004810 0x7fbb04004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 -- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb04005290 msgr2=0x7fbb0400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb04005290 0x7fbb0400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1de00640 1 -- 172.21.15.81:0/1572837138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb04003530 con 0x7fbb04003c40 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb1e800640 1 -- 172.21.15.81:0/1572837138 reap_dead start 2024-05-27T06:45:09.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.742+0000 7fbb17e00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04004810 0x7fbb04004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:09.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb1de00640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 0x7fbb04004090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009ab0 tx=0x7fbb1001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c736fdd8b9ce86ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb100096b0 con 0x7fbb04003c40 2024-05-27T06:45:09.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb100a6430 con 0x7fbb04003c40 2024-05-27T06:45:09.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb1d400640 1 -- 172.21.15.81:0/1572837138 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb100a6820 con 0x7fbb04003c40 2024-05-27T06:45:09.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 -- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 msgr2=0x7fbb04004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 0x7fbb04004090 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009ab0 tx=0x7fbb1001ca50 comp rx=0 tx=0).stop 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 -- 172.21.15.81:0/1572837138 shutdown_connections 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 --2- 172.21.15.81:0/1572837138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb04003c40 0x7fbb04004090 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 -- 172.21.15.81:0/1572837138 >> 172.21.15.81:0/1572837138 conn(0x7fbb20070700 msgr2=0x7fbb200cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 -- 172.21.15.81:0/1572837138 shutdown_connections 2024-05-27T06:45:09.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.746+0000 7fbb25200640 1 -- 172.21.15.81:0/1572837138 wait complete. 2024-05-27T06:45:09.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 Processor -- start 2024-05-27T06:45:09.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 -- start start 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04003c40 0x7fbb20145290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb201467f0 0x7fbb2014c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb200c7dd0 con 0x7fbb200ca190 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbb200c75c0 con 0x7fbb201467f0 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb25200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fbb200c5610 con 0x7fbb04003c40 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57200/0 (socket says 172.21.15.81:57200) 2024-05-27T06:45:09.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 -- 172.21.15.81:0/3952077360 learned_addr learned my addr 172.21.15.81:0/3952077360 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1de00640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04003c40 0x7fbb20145290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb17e00640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb201467f0 0x7fbb2014c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 -- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04003c40 msgr2=0x7fbb20145290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04003c40 0x7fbb20145290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 -- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb201467f0 msgr2=0x7fbb2014c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb201467f0 0x7fbb2014c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 -- 172.21.15.81:0/3952077360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb201457d0 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb1e800640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbb20073cf0 tx=0x7fbb18007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.750+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800aa80 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb18018070 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800ae60 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2014b430 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2014b9b0 con 0x7fbb200ca190 2024-05-27T06:45:09.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb2006a560 con 0x7fbb200ca190 2024-05-27T06:45:09.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbb1801d460 con 0x7fbb200ca190 2024-05-27T06:45:09.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb16000640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 0x7fbafc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:09.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb1808f830 con 0x7fbb200ca190 2024-05-27T06:45:09.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.754+0000 7fbb1de00640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 0x7fbafc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:09.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.758+0000 7fbb1de00640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 0x7fbafc071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb100ad000 tx=0x7fbb100046d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:09.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.758+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fbb1805c690 con 0x7fbb200ca190 2024-05-27T06:45:09.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.918+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb200c5940 con 0x7fbb200ca190 2024-05-27T06:45:09.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.918+0000 7fbb16000640 1 -- 172.21.15.81:0/3952077360 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7fbb1805bf90 con 0x7fbb200ca190 2024-05-27T06:45:09.915 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:09.915 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:03.608328+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 msgr2=0x7fbafc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 0x7fbafc071910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb100ad000 tx=0x7fbb100046d0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 msgr2=0x7fbb200ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbb20073cf0 tx=0x7fbb18007b10 comp rx=0 tx=0).stop 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 shutdown_connections 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbafc06f400 0x7fbafc071910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb200ca190 0x7fbb200ca5a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb201467f0 0x7fbb2014c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 --2- 172.21.15.81:0/3952077360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbb04003c40 0x7fbb20145290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:09.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 >> 172.21.15.81:0/3952077360 conn(0x7fbb20070700 msgr2=0x7fbb200c81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:09.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 shutdown_connections 2024-05-27T06:45:09.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:09.922+0000 7fbb25200640 1 -- 172.21.15.81:0/3952077360 wait complete. 2024-05-27T06:45:09.936 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-27T06:45:09.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:10.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 Processor -- start 2024-05-27T06:45:10.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 -- start start 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f82d40cfdb0 0x7f82d40d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f82d4154200 0x7f82d4154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d40c78b0 con 0x7f82d40ca770 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f82d4154b50 con 0x7f82d4154200 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d9400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f82d40c6ee0 con 0x7f82d40cfdb0 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f82d4154200 0x7f82d4154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f82d4154200 0x7f82d4154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57202/0 (socket says 172.21.15.81:57202) 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82d3200640 1 -- 172.21.15.81:0/471826949 learned_addr learned my addr 172.21.15.81:0/471826949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.090+0000 7f82c9e00640 1 --2- 172.21.15.81:0/471826949 >> v2:172.21.15.106:3300/0 conn(0x7f82d40cfdb0 0x7f82d40d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d3200640 1 -- 172.21.15.81:0/471826949 >> v2:172.21.15.106:3300/0 conn(0x7f82d40cfdb0 msgr2=0x7f82d40d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d3200640 1 --2- 172.21.15.81:0/471826949 >> v2:172.21.15.106:3300/0 conn(0x7f82d40cfdb0 0x7f82d40d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d3200640 1 -- 172.21.15.81:0/471826949 >> v1:172.21.15.106:6789/0 conn(0x7f82d40ca770 legacy=0x7f82d40cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d3200640 1 -- 172.21.15.81:0/471826949 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d40c71c0 con 0x7f82d4154200 2024-05-27T06:45:10.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d3200640 1 --2- 172.21.15.81:0/471826949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d4154200 0x7f82d4154610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82d4073180 tx=0x7f82cc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796203513e096ed0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d1e00640 1 -- 172.21.15.81:0/471826949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0a57d0 con 0x7f82d4154200 2024-05-27T06:45:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d1e00640 1 -- 172.21.15.81:0/471826949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82cc0a5970 con 0x7f82d4154200 2024-05-27T06:45:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d1e00640 1 -- 172.21.15.81:0/471826949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc0a5d40 con 0x7f82d4154200 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d9400640 1 -- 172.21.15.81:0/471826949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d4154200 msgr2=0x7f82d4154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.094+0000 7f82d9400640 1 --2- 172.21.15.81:0/471826949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d4154200 0x7f82d4154610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f82d4073180 tx=0x7f82cc01c9f0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- 172.21.15.81:0/471826949 shutdown_connections 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 --2- 172.21.15.81:0/471826949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d4154200 0x7f82d4154610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 --2- 172.21.15.81:0/471826949 >> v2:172.21.15.106:3300/0 conn(0x7f82d40cfdb0 0x7f82d40d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- 172.21.15.81:0/471826949 >> 172.21.15.81:0/471826949 conn(0x7f82d4068ae0 msgr2=0x7f82d4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- 172.21.15.81:0/471826949 shutdown_connections 2024-05-27T06:45:10.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- 172.21.15.81:0/471826949 wait complete. 2024-05-27T06:45:10.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 Processor -- start 2024-05-27T06:45:10.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- start start 2024-05-27T06:45:10.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82d40cfdb0 0x7f82d41564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d4154200 0x7f82d414f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82d40cc050 con 0x7f82d40ca770 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82d40cbed0 con 0x7f82d4154200 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d9400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f82d4167590 con 0x7f82d40cfdb0 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57212/0 (socket says 172.21.15.81:57212) 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.098+0000 7f82d2800640 1 -- 172.21.15.81:0/88944264 learned_addr learned my addr 172.21.15.81:0/88944264 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d3200640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d4154200 0x7f82d414f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82c9e00640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82d40cfdb0 0x7f82d41564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 -- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82d40cfdb0 msgr2=0x7f82d41564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82d40cfdb0 0x7f82d41564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 -- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d4154200 msgr2=0x7f82d414f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d4154200 0x7f82d414f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 -- 172.21.15.81:0/88944264 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d4157b50 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d2800640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f82c400c450 tx=0x7f82c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c40052d0 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c4005470 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d41f6c10 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c4012850 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d41f71d0 con 0x7f82d40ca770 2024-05-27T06:45:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d406a560 con 0x7f82d40ca770 2024-05-27T06:45:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.102+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f82c40129f0 con 0x7f82d40ca770 2024-05-27T06:45:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.106+0000 7f82cb400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 0x7f82b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.106+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f82c408ed00 con 0x7f82d40ca770 2024-05-27T06:45:10.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.106+0000 7f82c9e00640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 0x7f82b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.106+0000 7f82c9e00640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 0x7f82b0071910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82d4150980 tx=0x7f82bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.110+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f82c405b620 con 0x7f82d40ca770 2024-05-27T06:45:10.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.266+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82d40ccfd0 con 0x7f82d40ca770 2024-05-27T06:45:10.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82cb400640 1 -- 172.21.15.81:0/88944264 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f82c405f920 con 0x7f82d40ca770 2024-05-27T06:45:10.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:10.264 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:03.608328+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 msgr2=0x7f82b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 0x7f82b0071910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f82d4150980 tx=0x7f82bc008040 comp rx=0 tx=0).stop 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 msgr2=0x7f82d41f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f82c400c450 tx=0x7f82c400c910 comp rx=0 tx=0).stop 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 shutdown_connections 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82d40cfdb0 0x7f82d41564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f82b006f400 0x7f82b0071910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d4154200 0x7f82d414f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 --2- 172.21.15.81:0/88944264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82d40ca770 0x7f82d41f3500 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 >> 172.21.15.81:0/88944264 conn(0x7f82d4068ae0 msgr2=0x7f82d415e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 shutdown_connections 2024-05-27T06:45:10.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.270+0000 7f82d9400640 1 -- 172.21.15.81:0/88944264 wait complete. 2024-05-27T06:45:10.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-27T06:45:10.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-27T06:45:10.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-27T06:45:10.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-27T06:45:10.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-27T06:45:10.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-27T06:45:10.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-27T06:45:10.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-27T06:45:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.558+0000 7f61b0e00640 1 Processor -- start 2024-05-27T06:45:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.558+0000 7f61b0e00640 1 -- start start 2024-05-27T06:45:10.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f778fc00640 1 Processor -- start 2024-05-27T06:45:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f61b0e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c9b40 0x7f61ac0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f61b0e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f61ac0c70e0 0x7f61ac15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f61b0e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61ac0c4ba0 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f61b0e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f61ac1587e0 con 0x7f61ac0c70e0 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.562+0000 7f61b0e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f61ac158ac0 con 0x7f61ac0c9b40 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f61ac0c6cd0 0x7f61ac06f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59074/0 (socket says 172.21.15.81:59074) 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 -- 172.21.15.81:0/1257186934 learned_addr learned my addr 172.21.15.81:0/1257186934 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61abe00640 1 --2- 172.21.15.81:0/1257186934 >> v2:172.21.15.81:3300/0 conn(0x7f61ac0c70e0 0x7f61ac15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778fc00640 1 -- start start 2024-05-27T06:45:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706797361 0 0) 0x7f61ac0c4ba0 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6190003530 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 68958002 0 0) 0x7f6190003530 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c9b40 msgr2=0x7f61ac0c9310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T06:45:10.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac0c9b40 0x7f61ac0c9310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 >> v2:172.21.15.81:3300/0 conn(0x7f61ac0c70e0 msgr2=0x7f61ac15bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 --2- 172.21.15.81:0/1257186934 >> v2:172.21.15.81:3300/0 conn(0x7f61ac0c70e0 0x7f61ac15bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac158da0 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.1 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f61a4001a80 con 0x7f61ac0c6cd0 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 >> v1:172.21.15.106:6789/0 conn(0x7f61ac0c6cd0 legacy=0x7f61ac06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6190003a60 0x7f6190003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6190004810 0x7f6190004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190005280 0x7f619000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac1587e0 con 0x7f6190003a60 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac158ac0 con 0x7f6190005280 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f619000c210 con 0x7f6190004810 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61ab400640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6190003a60 0x7f6190003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6190004810 0x7f6190004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61abe00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190005280 0x7f619000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778fc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77800457e0 0x7f778004b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778fc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77800e7960 con 0x7f77800dda20 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778fc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77800e4620 con 0x7f778004bbd0 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778fc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f77800dde30 con 0x7f77800457e0 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 -- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190005280 msgr2=0x7f619000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190005280 0x7f619000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 -- 172.21.15.81:0/1257186934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6190003a60 msgr2=0x7f6190003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6190003a60 0x7f6190003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f61aaa00640 1 -- 172.21.15.81:0/1257186934 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6190003530 con 0x7f6190004810 2024-05-27T06:45:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f76fb600640 1 Processor -- start 2024-05-27T06:45:10.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f76fb600640 1 -- start start 2024-05-27T06:45:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7fb019800640 1 Processor -- start 2024-05-27T06:45:10.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7fb019800640 1 -- start start 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778dc00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f778004bbd0 0x7f77800ee040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59090/0 (socket says 172.21.15.81:59090) 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.566+0000 7f778dc00640 1 -- 172.21.15.81:0/3632592473 learned_addr learned my addr 172.21.15.81:0/3632592473 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 --2- 172.21.15.81:0/3632592473 >> v2:172.21.15.81:3300/0 conn(0x7f77800457e0 0x7f778004b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741132318 0 0) 0x7f77800e4620 con 0x7f778004bbd0 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7770003530 con 0x7f778004bbd0 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1426354692 0 0) 0x7f7770003530 con 0x7f778004bbd0 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 >> v2:172.21.15.81:3300/0 conn(0x7f77800457e0 msgr2=0x7f778004b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 --2- 172.21.15.81:0/3632592473 >> v2:172.21.15.81:3300/0 conn(0x7f77800457e0 0x7f778004b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 >> v1:172.21.15.81:6789/0 conn(0x7f77800dda20 legacy=0x7f77800060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77800e3860 con 0x7f778004bbd0 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.1 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f7788078030 con 0x7f778004bbd0 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 >> v1:172.21.15.106:6789/0 conn(0x7f778004bbd0 legacy=0x7f77800ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77700039c0 0x7f7770003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 0x7f7770004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7770005160 0x7f777000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77800dde30 con 0x7f77700039c0 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f777000bfc0 con 0x7f7770005160 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f777000c1c0 con 0x7f7770004810 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f778d200640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77700039c0 0x7f7770003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 0x7f7770004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f778dc00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7770005160 0x7f777000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 -- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7770005160 msgr2=0x7f777000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7770005160 0x7f777000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 -- 172.21.15.81:0/3632592473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77700039c0 msgr2=0x7f7770003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77700039c0 0x7f7770003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f7787e00640 1 -- 172.21.15.81:0/3632592473 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77800e3860 con 0x7f7770004810 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f778dc00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7770005160 0x7f777000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:10.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f778dc00640 1 -- 172.21.15.81:0/3632592473 reap_dead start 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61aaa00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6190004810 0x7f6190004c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f61a400a3d0 tx=0x7f61a4023e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d13761a9d11bd59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76fb600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f76ec0dfdf0 0x7f76ec0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76fb600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec007920 0x7f76ec0e31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76fb600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76ec04a0e0 con 0x7f76ec0e0200 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76fb600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f76ec050db0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76fb600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76ec0df630 con 0x7f76ec007920 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76f3e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f76ec0e0200 0x7f76ec0e2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59096/0 (socket says 172.21.15.81:59096) 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76f3e00640 1 -- 172.21.15.81:0/128699686 learned_addr learned my addr 172.21.15.81:0/128699686 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76f8c00640 1 --2- 172.21.15.81:0/128699686 >> v2:172.21.15.106:3300/0 conn(0x7f76ec0dfdf0 0x7f76ec0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7f76f9600640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec007920 0x7f76ec0e31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f3400640 1 -- 172.21.15.81:0/128699686 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456354308 0 0) 0x7f76ec04a0e0 con 0x7f76ec0e0200 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f3400640 1 -- 172.21.15.81:0/128699686 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76dc003530 con 0x7f76ec0e0200 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f8c00640 1 -- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec007920 msgr2=0x7f76ec0e31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f8c00640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec007920 0x7f76ec0e31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f8c00640 1 -- 172.21.15.81:0/128699686 >> v1:172.21.15.106:6789/0 conn(0x7f76ec0e0200 legacy=0x7f76ec0e2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f76f8c00640 1 -- 172.21.15.81:0/128699686 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec04a380 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7fb019800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb014159390 0x7fb0140c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7fb019800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 0x7fb01406fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7fb019800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0140c5880 con 0x7fb0140c9d90 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7fb019800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb0140c41d0 con 0x7fb014159390 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.570+0000 7fb019800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb0140c4490 con 0x7fb0140c9370 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb012a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb0140c9d90 0x7fb014152ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37036/0 (socket says 172.21.15.81:37036) 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb012a00640 1 -- 172.21.15.81:0/779743202 learned_addr learned my addr 172.21.15.81:0/779743202 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb012000640 1 --2- 172.21.15.81:0/779743202 >> v2:172.21.15.106:3300/0 conn(0x7fb014159390 0x7fb0140c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a400a410 con 0x7f6190004810 2024-05-27T06:45:10.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a40b1020 con 0x7f6190004810 2024-05-27T06:45:10.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb013400640 1 --2- 172.21.15.81:0/779743202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 0x7fb01406fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61aa000640 1 -- 172.21.15.81:0/1257186934 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40adac0 con 0x7f6190004810 2024-05-27T06:45:10.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f1e56c00640 1 Processor -- start 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb013400640 1 -- 172.21.15.81:0/779743202 >> v2:172.21.15.106:3300/0 conn(0x7fb014159390 msgr2=0x7fb0140c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb013400640 1 --2- 172.21.15.81:0/779743202 >> v2:172.21.15.106:3300/0 conn(0x7fb014159390 0x7fb0140c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb013400640 1 -- 172.21.15.81:0/779743202 >> v1:172.21.15.81:6789/0 conn(0x7fb0140c9d90 legacy=0x7fb014152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fa675000640 1 Processor -- start 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 -- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6190004810 msgr2=0x7f6190004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 --2- 172.21.15.81:0/1257186934 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6190004810 0x7f6190004c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f61a400a3d0 tx=0x7f61a4023e00 comp rx=0 tx=0).stop 2024-05-27T06:45:10.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fb013400640 1 -- 172.21.15.81:0/779743202 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0140c4770 con 0x7fb0140c9370 2024-05-27T06:45:10.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f1e56c00640 1 -- start start 2024-05-27T06:45:10.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61abe00640 1 -- 172.21.15.81:0/1257186934 reap_dead start 2024-05-27T06:45:10.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 -- 172.21.15.81:0/1257186934 shutdown_connections 2024-05-27T06:45:10.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 -- 172.21.15.81:0/1257186934 >> 172.21.15.81:0/1257186934 conn(0x7f61ac06d150 msgr2=0x7f61ac0cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7fa675000640 1 -- start start 2024-05-27T06:45:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 -- 172.21.15.81:0/1257186934 shutdown_connections 2024-05-27T06:45:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.574+0000 7f61b0e00640 1 -- 172.21.15.81:0/1257186934 wait complete. 2024-05-27T06:45:10.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76f8c00640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0dfdf0 0x7f76ec0073e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76f406a2d0 tx=0x7f76f40a30f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69e4324161afc04c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f7787e00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 0x7f7770004c20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f777400ba30 tx=0x7f777401c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bacbcc96c89d3572 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76f3400640 1 -- 172.21.15.81:0/128699686 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4131640 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76f3400640 1 -- 172.21.15.81:0/128699686 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f41317e0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76f3400640 1 -- 172.21.15.81:0/128699686 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4131bf0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740b3070 con 0x7f7770004810 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa675000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa670156760 0x7fa67015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa675000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa67006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1e50073970 0x7f1e50073d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e500c7320 con 0x7f1e500ce350 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1e500c7600 con 0x7f1e50073970 2024-05-27T06:45:10.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e500723e0 con 0x7f1e500c77d0 2024-05-27T06:45:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f61bf200640 1 Processor -- start 2024-05-27T06:45:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa675000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67015d5f0 con 0x7fa6700b5bb0 2024-05-27T06:45:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa675000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa670156b70 con 0x7fa670156760 2024-05-27T06:45:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa675000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa6700cd550 con 0x7fa67006cb40 2024-05-27T06:45:10.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa670156760 0x7fa67015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f61bf200640 1 -- start start 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 -- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0dfdf0 msgr2=0x7f76ec0073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0dfdf0 0x7f76ec0073e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76f406a2d0 tx=0x7f76f40a30f0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 -- 172.21.15.81:0/128699686 shutdown_connections 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec007920 0x7f76ec0e31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 --2- 172.21.15.81:0/128699686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0dfdf0 0x7f76ec0073e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 -- 172.21.15.81:0/128699686 >> 172.21.15.81:0/128699686 conn(0x7f76ec00e390 msgr2=0x7f76ec00e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e54e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e54e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54842/0 (socket says 172.21.15.81:54842) 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e55800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1e500ce350 0x7f1e5015d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37040/0 (socket says 172.21.15.81:37040) 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e55800640 1 -- 172.21.15.81:0/2614967565 learned_addr learned my addr 172.21.15.81:0/2614967565 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77740ae630 con 0x7f7770004810 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 -- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 msgr2=0x7f7770004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66e800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa6700b5bb0 0x7fa6700b5fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37042/0 (socket says 172.21.15.81:37042) 2024-05-27T06:45:10.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66e800640 1 -- 172.21.15.81:0/1845054849 learned_addr learned my addr 172.21.15.81:0/1845054849 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f61b0e00640 1 Processor -- start 2024-05-27T06:45:10.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa670156760 0x7fa67015d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57236/0 (socket says 172.21.15.81:57236) 2024-05-27T06:45:10.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f7787400640 1 -- 172.21.15.81:0/3632592473 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740b3070 con 0x7f7770004810 2024-05-27T06:45:10.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f61b0e00640 1 -- start start 2024-05-27T06:45:10.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 0x7f7770004c20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f777400ba30 tx=0x7f777401c6a0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 -- 172.21.15.81:0/3632592473 shutdown_connections 2024-05-27T06:45:10.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e56200640 1 --2- 172.21.15.81:0/2614967565 >> v2:172.21.15.81:3300/0 conn(0x7f1e50073970 0x7f1e50073d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 --2- 172.21.15.81:0/3632592473 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7770004810 0x7f7770004c20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 -- 172.21.15.81:0/3632592473 >> 172.21.15.81:0/3632592473 conn(0x7f77800503c0 msgr2=0x7f77800507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66d400640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa67006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 -- 172.21.15.81:0/1845054849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 msgr2=0x7fa67006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa67006cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 -- 172.21.15.81:0/1845054849 >> v1:172.21.15.81:6789/0 conn(0x7fa6700b5bb0 legacy=0x7fa6700b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fa66de00640 1 -- 172.21.15.81:0/1845054849 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6700cc660 con 0x7fa670156760 2024-05-27T06:45:10.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 -- 172.21.15.81:0/3632592473 shutdown_connections 2024-05-27T06:45:10.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f778fc00640 1 -- 172.21.15.81:0/3632592473 wait complete. 2024-05-27T06:45:10.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 -- 172.21.15.81:0/128699686 shutdown_connections 2024-05-27T06:45:10.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f76fb600640 1 -- 172.21.15.81:0/128699686 wait complete. 2024-05-27T06:45:10.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7fb013400640 1 --2- 172.21.15.81:0/779743202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 0x7fb01406fda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009d90 tx=0x7fb00c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c678cbf131a2b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e54e00640 1 -- 172.21.15.81:0/2614967565 >> v2:172.21.15.81:3300/0 conn(0x7f1e50073970 msgr2=0x7f1e50073d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f1e54e00640 1 --2- 172.21.15.81:0/2614967565 >> v2:172.21.15.81:3300/0 conn(0x7f1e50073970 0x7f1e50073d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f1e54e00640 1 -- 172.21.15.81:0/2614967565 >> v1:172.21.15.81:6789/0 conn(0x7f1e500ce350 legacy=0x7f1e5015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f1e54e00640 1 -- 172.21.15.81:0/2614967565 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5015ca10 con 0x7f1e500c77d0 2024-05-27T06:45:10.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb011600640 1 -- 172.21.15.81:0/779743202 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0b2070 con 0x7fb0140c9370 2024-05-27T06:45:10.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb011600640 1 -- 172.21.15.81:0/779743202 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c0a5830 con 0x7fb0140c9370 2024-05-27T06:45:10.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb011600640 1 -- 172.21.15.81:0/779743202 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0a5c00 con 0x7fb0140c9370 2024-05-27T06:45:10.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.578+0000 7f8255000640 1 Processor -- start 2024-05-27T06:45:10.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f8255000640 1 -- start start 2024-05-27T06:45:10.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190004810 0x7f61ac159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 -- 172.21.15.81:0/779743202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 msgr2=0x7fb01406fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 --2- 172.21.15.81:0/779743202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 0x7fb01406fda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009d90 tx=0x7fb00c01c970 comp rx=0 tx=0).stop 2024-05-27T06:45:10.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 0x7f61ac1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bf200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61c0165fa0 0x7f61c01683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61ac14f030 0x7f61ac146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bf200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c00c7200 con 0x7f61c00c6a00 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61ab400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190004810 0x7f61ac159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 0x7f61ac1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61ab400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190004810 0x7f61ac159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46834/0 (socket says 172.21.15.81:46834) 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 0x7f61ac1504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54860/0 (socket says 172.21.15.81:54860) 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac070020 con 0x7f61ac14f030 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac159360 con 0x7f6190004810 2024-05-27T06:45:10.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61b0e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f61ac0701a0 con 0x7f61ac1e39f0 2024-05-27T06:45:10.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bde00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f61c01567f0 0x7f61c0160380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59110/0 (socket says 172.21.15.81:59110) 2024-05-27T06:45:10.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bde00640 1 -- 172.21.15.81:0/1215363045 learned_addr learned my addr 172.21.15.81:0/1215363045 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c015d340 con 0x7f61c01567f0 2024-05-27T06:45:10.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f61c00b47e0 con 0x7f61c0165fa0 2024-05-27T06:45:10.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61be800640 1 --2- 172.21.15.81:0/1215363045 >> v2:172.21.15.106:3300/0 conn(0x7f61c0165fa0 0x7f61c01683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 -- 172.21.15.81:0/779743202 shutdown_connections 2024-05-27T06:45:10.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 --2- 172.21.15.81:0/779743202 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb0140c9370 0x7fb01406fda0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 --2- 172.21.15.81:0/779743202 >> v2:172.21.15.106:3300/0 conn(0x7fb014159390 0x7fb0140c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 -- 172.21.15.81:0/779743202 >> 172.21.15.81:0/779743202 conn(0x7fb01406cd90 msgr2=0x7fb0140cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61ab400640 1 -- 172.21.15.81:0/3369081120 learned_addr learned my addr 172.21.15.81:0/3369081120 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61abe00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61ac14f030 0x7f61ac146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 -- 172.21.15.81:0/3369081120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190004810 msgr2=0x7f61ac159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6190004810 0x7f61ac159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 -- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61ac14f030 msgr2=0x7f61ac146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61ac14f030 0x7f61ac146160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61aaa00640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac150c40 con 0x7f61ac1e39f0 2024-05-27T06:45:10.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 -- 172.21.15.81:0/779743202 shutdown_connections 2024-05-27T06:45:10.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f76fb600640 1 Processor -- start 2024-05-27T06:45:10.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61be800640 1 -- 172.21.15.81:0/1215363045 >> v1:172.21.15.106:6789/0 conn(0x7f61c01567f0 legacy=0x7f61c0160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61be800640 1 -- 172.21.15.81:0/1215363045 >> v1:172.21.15.81:6789/0 conn(0x7f61c00c6a00 legacy=0x7f61c00d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f61be800640 1 -- 172.21.15.81:0/1215363045 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c00cd230 con 0x7f61c0165fa0 2024-05-27T06:45:10.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7f76fb600640 1 -- start start 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.582+0000 7fb019800640 1 -- 172.21.15.81:0/779743202 wait complete. 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f8255000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 0x7f82480d6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f8255000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 0x7f8248046e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f8255000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8248047520 con 0x7f82480dbfc0 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f8255000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f8248045360 con 0x7f8248046a20 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f8255000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8248045590 con 0x7f82480063c0 2024-05-27T06:45:10.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61aaa00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 0x7f61ac1504f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009000 tx=0x7f6190005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 Processor -- start 2024-05-27T06:45:10.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4004d70 con 0x7f61ac1e39f0 2024-05-27T06:45:10.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a40ad8c0 con 0x7f61ac1e39f0 2024-05-27T06:45:10.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40adcb0 con 0x7f61ac1e39f0 2024-05-27T06:45:10.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824fe00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f82480dbfc0 0x7f82480dc3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59120/0 (socket says 172.21.15.81:59120) 2024-05-27T06:45:10.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824fe00640 1 -- 172.21.15.81:0/3652112367 learned_addr learned my addr 172.21.15.81:0/3652112367 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824ea00640 1 --2- 172.21.15.81:0/3652112367 >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 0x7f8248046e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 -- start start 2024-05-27T06:45:10.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f1e54e00640 1 --2- 172.21.15.81:0/2614967565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097c0 tx=0x7f1e3c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee3b7c4b4ddb1c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824f400640 1 --2- 172.21.15.81:0/3652112367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 0x7f82480d6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61ac14fd60 con 0x7f61ac1e39f0 2024-05-27T06:45:10.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61ac14a760 con 0x7f61ac1e39f0 2024-05-27T06:45:10.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7fa66de00640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67015d0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097c0 tx=0x7fa65c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8072e517ef2354 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824e000640 1 -- 172.21.15.81:0/3652112367 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048312858 0 0) 0x7f8248047520 con 0x7f82480dbfc0 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824e000640 1 -- 172.21.15.81:0/3652112367 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8238003530 con 0x7f82480dbfc0 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76fb600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec007920 0x7f76ec0d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f1e4fe00640 1 -- 172.21.15.81:0/2614967565 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0b1070 con 0x7f1e500c77d0 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f1e4fe00640 1 -- 172.21.15.81:0/2614967565 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3c0ac410 con 0x7f1e500c77d0 2024-05-27T06:45:10.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f1e4fe00640 1 -- 172.21.15.81:0/2614967565 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c0ac7e0 con 0x7f1e500c77d0 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7770004810 0x7f77800dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 0x7f77800e5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77800cfe90 0x7f77800d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61ac050870 con 0x7f61ac1e39f0 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76fb600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec0dfdf0 0x7f76ec0ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76fb600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0e0200 0x7f76ec0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76fb600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec050a20 con 0x7f76ec007920 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76f8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec007920 0x7f76ec0d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76f8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec007920 0x7f76ec0d4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57264/0 (socket says 172.21.15.81:57264) 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76f8c00640 1 -- 172.21.15.81:0/1518898674 learned_addr learned my addr 172.21.15.81:0/1518898674 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7fa66ca00640 1 -- 172.21.15.81:0/1845054849 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0b2070 con 0x7fa670156760 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7fa66ca00640 1 -- 172.21.15.81:0/1845054849 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c0ad640 con 0x7fa670156760 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7fa66ca00640 1 -- 172.21.15.81:0/1845054849 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0ada10 con 0x7fa670156760 2024-05-27T06:45:10.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824f400640 1 -- 172.21.15.81:0/3652112367 >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 msgr2=0x7f8248046e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824f400640 1 --2- 172.21.15.81:0/3652112367 >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 0x7f8248046e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824f400640 1 -- 172.21.15.81:0/3652112367 >> v1:172.21.15.106:6789/0 conn(0x7f82480dbfc0 legacy=0x7f82480dc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824f400640 1 -- 172.21.15.81:0/3652112367 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82480358f0 con 0x7f82480063c0 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f824ea00640 1 --2- 172.21.15.81:0/3652112367 >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 0x7f8248046e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77800e36c0 con 0x7f7770004810 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77800e7c20 con 0x7f77800cfe90 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f778fc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f77800e7670 con 0x7f77800d9a30 2024-05-27T06:45:10.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f7787e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 0x7f77800e5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 0x7f77800e5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54886/0 (socket says 172.21.15.81:54886) 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 -- 172.21.15.81:0/3549272926 learned_addr learned my addr 172.21.15.81:0/3549272926 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.586+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec0d9be0 con 0x7f76ec0e0200 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f9600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0e0200 0x7f76ec0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76ec04f0b0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec0dfdf0 0x7f76ec0ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fb019800640 1 Processor -- start 2024-05-27T06:45:10.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fb019800640 1 -- start start 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 -- 172.21.15.81:0/2614967565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 msgr2=0x7f1e50159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 --2- 172.21.15.81:0/2614967565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0097c0 tx=0x7f1e3c01c4c0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 -- 172.21.15.81:0/2614967565 shutdown_connections 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 --2- 172.21.15.81:0/2614967565 >> v2:172.21.15.81:3300/0 conn(0x7f1e50073970 0x7f1e50073d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 --2- 172.21.15.81:0/2614967565 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500c77d0 0x7f1e50159fd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 -- 172.21.15.81:0/2614967565 >> 172.21.15.81:0/2614967565 conn(0x7f1e5006cd90 msgr2=0x7f1e500cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 -- 172.21.15.81:0/1845054849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 msgr2=0x7fa67015d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67015d0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0097c0 tx=0x7fa65c01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 -- 172.21.15.81:0/3549272926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77800cfe90 msgr2=0x7f77800d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77800cfe90 0x7f77800d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 -- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7770004810 msgr2=0x7f77800dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7770004810 0x7f77800dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77800e0400 con 0x7f77800d9a30 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61be800640 1 --2- 172.21.15.81:0/1215363045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c0165fa0 0x7f61c01683c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61b00097c0 tx=0x7f61b001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6365db3fdaa48d0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 -- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0e0200 msgr2=0x7f76ec0ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76ec0e0200 0x7f76ec0ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 -- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec007920 msgr2=0x7f76ec0d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76ec007920 0x7f76ec0d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec0d4f60 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bca00640 1 -- 172.21.15.81:0/1215363045 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b00b0070 con 0x7f61c0165fa0 2024-05-27T06:45:10.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 -- 172.21.15.81:0/2614967565 shutdown_connections 2024-05-27T06:45:10.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bca00640 1 -- 172.21.15.81:0/1215363045 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b00ab600 con 0x7f61c0165fa0 2024-05-27T06:45:10.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bca00640 1 -- 172.21.15.81:0/1215363045 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b00ab9d0 con 0x7f61c0165fa0 2024-05-27T06:45:10.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7787e00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 0x7f77800e5cb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77740b1040 tx=0x7f7770005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f3e00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec0dfdf0 0x7f76ec0ca0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76ec04a2b0 tx=0x7f76e000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f61a40070d0 con 0x7f61ac1e39f0 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 -- 172.21.15.81:0/1845054849 shutdown_connections 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa67006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 --2- 172.21.15.81:0/1845054849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67015d0b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fa675000640 1 -- 172.21.15.81:0/1845054849 >> 172.21.15.81:0/1845054849 conn(0x7fa6700cecc0 msgr2=0x7fa6700d1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f1e56c00640 1 -- 172.21.15.81:0/2614967565 wait complete. 2024-05-27T06:45:10.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c0165fa0 msgr2=0x7f61c01683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 --2- 172.21.15.81:0/1215363045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c0165fa0 0x7f61c01683c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f61b00097c0 tx=0x7f61b001c410 comp rx=0 tx=0).stop 2024-05-27T06:45:10.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740bd020 con 0x7f77800d9a30 2024-05-27T06:45:10.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7774004220 con 0x7f77800d9a30 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740aee60 con 0x7f77800d9a30 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0017070 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e0005cf0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e00072d0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ec0d8810 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec0d8d90 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f778fc00640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77800d4c40 con 0x7f77800d9a30 2024-05-27T06:45:10.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f778fc00640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77800de640 con 0x7f77800d9a30 2024-05-27T06:45:10.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fb019800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 0x7fb0141546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7fb019800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0140c9d90 0x7fb0140ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f824f400640 1 --2- 172.21.15.81:0/3652112367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 0x7f82480d6040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f825007dba0 tx=0x7f825009c700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=525960298a0c6390 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 shutdown_connections 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 --2- 172.21.15.81:0/1215363045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c0165fa0 0x7f61c01683c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 >> 172.21.15.81:0/1215363045 conn(0x7f61c006e790 msgr2=0x7f61c006eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 shutdown_connections 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61bf200640 1 -- 172.21.15.81:0/1215363045 wait complete. 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fa675000640 1 -- 172.21.15.81:0/1845054849 shutdown_connections 2024-05-27T06:45:10.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 0x7fb0141546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 Processor -- start 2024-05-27T06:45:10.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 0x7fb0141546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57290/0 (socket says 172.21.15.81:57290) 2024-05-27T06:45:10.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f61a3400640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f618806f400 0x7f6188071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fa675000640 1 -- 172.21.15.81:0/1845054849 wait complete. 2024-05-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f824e000640 1 -- 172.21.15.81:0/3652112367 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250131070 con 0x7f82480063c0 2024-05-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.590+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f76d0001040 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f824e000640 1 -- 172.21.15.81:0/3652112367 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825012c600 con 0x7f82480063c0 2024-05-27T06:45:10.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f824e000640 1 -- 172.21.15.81:0/3652112367 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825012ca30 con 0x7f82480063c0 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012a00640 1 -- 172.21.15.81:0/3861438388 learned_addr learned my addr 172.21.15.81:0/3861438388 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb019800640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb014159390 0x7fb01415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0140c9d90 0x7fb0140ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0140c9860 con 0x7fb0140c9370 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb0141597a0 con 0x7fb0140c9d90 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb0141565b0 con 0x7fb014159390 2024-05-27T06:45:10.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb013400640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb014159390 0x7fb01415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f61a4130f00 con 0x7f61ac1e39f0 2024-05-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61ab400640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f618806f400 0x7f6188071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61a3400640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] conn(0x7f6188077650 0x7f6188079aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61a40ade50 con 0x7f6188077650 2024-05-27T06:45:10.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f61a41312b0 con 0x7f61ac1e39f0 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 -- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb014159390 msgr2=0x7fb01415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb014159390 0x7fb01415a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 -- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 msgr2=0x7fb0141546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 0x7fb0141546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb014153d30 con 0x7fb0140c9d90 2024-05-27T06:45:10.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f775c001040 con 0x7f77800d9a30 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 -- 172.21.15.81:0/3652112367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 msgr2=0x7f82480d6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 --2- 172.21.15.81:0/3652112367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 0x7f82480d6040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f825007dba0 tx=0x7f825009c700 comp rx=0 tx=0).stop 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 -- start start 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50073970 0x7f1e50153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e500c77d0 0x7f1e50154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 0x7f1e5014cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50159580 con 0x7f1e50073970 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e50159400 con 0x7f1e500c77d0 2024-05-27T06:45:10.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5006f9c0 con 0x7f1e500ce350 2024-05-27T06:45:10.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012000640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0140c9d90 0x7fb0140ccf00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0140c79c0 tx=0x7fb00000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61abe00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] conn(0x7f6188077650 0x7f6188079aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb012a00640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0140c9370 0x7fb0141546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f76e001a070 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000017070 con 0x7fb0140c9d90 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00000c6d0 con 0x7fb0140c9d90 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00000caa0 con 0x7fb0140c9d90 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 -- 172.21.15.81:0/3652112367 shutdown_connections 2024-05-27T06:45:10.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 --2- 172.21.15.81:0/3652112367 >> v2:172.21.15.106:3300/0 conn(0x7f8248046a20 0x7f8248046e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 --2- 172.21.15.81:0/3652112367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f82480063c0 0x7f82480d6040 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f8255000640 1 -- 172.21.15.81:0/3652112367 >> 172.21.15.81:0/3652112367 conn(0x7f82480503c0 msgr2=0x7f82480507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 0x7f1e5014cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb014155ad0 con 0x7fb0140c9d90 2024-05-27T06:45:10.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 0x7f1e5014cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54902/0 (socket says 172.21.15.81:54902) 2024-05-27T06:45:10.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f1e56200640 1 -- 172.21.15.81:0/316338654 learned_addr learned my addr 172.21.15.81:0/316338654 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61abe00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] conn(0x7f6188077650 0x7f6188079aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e55800640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50073970 0x7f1e50153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb014156090 con 0x7fb0140c9d90 2024-05-27T06:45:10.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f77740c0070 con 0x7f77800d9a30 2024-05-27T06:45:10.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61ab400640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f618806f400 0x7f6188071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f619c000f50 tx=0x7f619c006ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61bf200640 1 Processor -- start 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.594+0000 7f61bf200640 1 -- start start 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c00c6a00 0x7f61c0159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 0x7f61c015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61c0165fa0 0x7f61c0158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61c0160a90 con 0x7f61c0165fa0 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f61c015d600 con 0x7f61c00c6a00 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bf200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f61c00cd930 con 0x7f61c01567f0 2024-05-27T06:45:10.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e56200640 1 -- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e500c77d0 msgr2=0x7f1e50154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e54e00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e500c77d0 0x7f1e50154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb014154010 con 0x7fb0140c9d90 2024-05-27T06:45:10.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e56200640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e500c77d0 0x7f1e50154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e56200640 1 -- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50073970 msgr2=0x7f1e50153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e56200640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50073970 0x7f1e50153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 0x7f61c015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 0x7f61c015ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54908/0 (socket says 172.21.15.81:54908) 2024-05-27T06:45:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 -- 172.21.15.81:0/1433509198 learned_addr learned my addr 172.21.15.81:0/1433509198 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== osd.1 v2:172.21.15.81:6800/404979354 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61a40ade50 con 0x7f6188077650 2024-05-27T06:45:10.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bde00640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c00c6a00 0x7f61c0159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7fa675000640 1 Processor -- start 2024-05-27T06:45:10.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f0c00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f76d406f500 0x7f76d4071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f8255000640 1 -- 172.21.15.81:0/3652112367 shutdown_connections 2024-05-27T06:45:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f8255000640 1 -- 172.21.15.81:0/3652112367 wait complete. 2024-05-27T06:45:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f776806f500 0x7f7768071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 -- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c00c6a00 msgr2=0x7f61c0159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61c00c6a00 0x7f61c0159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 -- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61c0165fa0 msgr2=0x7f61c0158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61c0165fa0 0x7f61c0158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c0153860 con 0x7f61c01567f0 2024-05-27T06:45:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f76e008e800 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7fa675000640 1 -- start start 2024-05-27T06:45:10.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7774128c30 con 0x7f77800d9a30 2024-05-27T06:45:10.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f0c00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] conn(0x7f76d40777d0 0x7f76d4079c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76e0014ec0 con 0x7f76d40777d0 2024-05-27T06:45:10.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f76e008ebf0 con 0x7f76ec0dfdf0 2024-05-27T06:45:10.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61bd400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 0x7f61c015ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00b410 tx=0x7f61ac00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] conn(0x7f77680777d0 0x7f7768079c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f776807a2f0 con 0x7f77680777d0 2024-05-27T06:45:10.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f777412b290 con 0x7f77800d9a30 2024-05-27T06:45:10.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f778d200640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f776806f500 0x7f7768071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f76f8c00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f76d406f500 0x7f76d4071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac015030 con 0x7f61c01567f0 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac004020 con 0x7f61c01567f0 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e56200640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e500c8f20 con 0x7f1e500ce350 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e54e00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e500c77d0 0x7f1e50154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.598+0000 7f1e55800640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50073970 0x7f1e50153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:10.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f76f9600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] conn(0x7f76d40777d0 0x7f76d4079c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb00001a070 con 0x7fb0140c9d90 2024-05-27T06:45:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e56200640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 0x7f1e5014cf10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e48007d40 tx=0x7f1e4800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f778dc00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] conn(0x7f77680777d0 0x7f7768079c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f778dc00640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] conn(0x7f77680777d0 0x7f7768079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014420 con 0x7f1e500ce350 2024-05-27T06:45:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e48019070 con 0x7f1e500ce350 2024-05-27T06:45:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014420 con 0x7f1e500ce350 2024-05-27T06:45:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f8255000640 1 Processor -- start 2024-05-27T06:45:10.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f8255000640 1 -- start start 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa670156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac004410 con 0x7f61c01567f0 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f61bf200640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c0157950 con 0x7f61c01567f0 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f61bf200640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c01570d0 con 0x7f61c01567f0 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faff406f4d0 0x7faff40719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb00008f530 con 0x7fb0140c9d90 2024-05-27T06:45:10.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] conn(0x7faff40777a0 0x7faff4079bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6700b5bb0 0x7fa670145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb00003ee50 con 0x7faff40777a0 2024-05-27T06:45:10.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb00008f920 con 0x7fb0140c9d90 2024-05-27T06:45:10.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e56c00640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e500c9200 con 0x7f1e500ce350 2024-05-27T06:45:10.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb013400640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] conn(0x7faff40777a0 0x7faff4079bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa66de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa670156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700b5720 con 0x7fa670156760 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa66de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa670156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54922/0 (socket says 172.21.15.81:54922) 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa66de00640 1 -- 172.21.15.81:0/670172851 learned_addr learned my addr 172.21.15.81:0/670172851 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700cd1e0 con 0x7fa6700b5bb0 2024-05-27T06:45:10.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fb012a00640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faff406f4d0 0x7faff40719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa6700c8c50 con 0x7fa67006cb40 2024-05-27T06:45:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7fa66d400640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6700b5bb0 0x7fa670145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f778d200640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f776806f500 0x7f7768071a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7780003ed0 tx=0x7f777c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f1e56c00640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e50155ca0 con 0x7f1e500ce350 2024-05-27T06:45:10.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f76f9600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] conn(0x7f76d40777d0 0x7f76d4079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.602+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== osd.0 v2:172.21.15.81:6822/1838218899 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f776807a2f0 con 0x7f77680777d0 2024-05-27T06:45:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 0x7f82480d7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8248046a20 0x7f8248179cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480dbfc0 0x7f8248049fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8248036190 con 0x7f82480dbfc0 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8248035eb0 con 0x7f82480063c0 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f8255000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8248045bd0 con 0x7f8248046a20 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 0x7f82480d7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f76f8c00640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f76d406f500 0x7f76d4071a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f76f40522c0 tx=0x7f76f406c380 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 0x7f82480d7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46916/0 (socket says 172.21.15.81:46916) 2024-05-27T06:45:10.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 -- 172.21.15.81:0/961733147 learned_addr learned my addr 172.21.15.81:0/961733147 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61c0159140 con 0x7f61c01567f0 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fb013400640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] conn(0x7faff40777a0 0x7faff4079bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fb012a00640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faff406f4d0 0x7faff40719e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb004007fd0 tx=0x7fb004009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824fe00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480dbfc0 0x7f8248049fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 -- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8248046a20 msgr2=0x7f8248179cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:45:10.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8248046a20 0x7f8248179cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 -- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480dbfc0 msgr2=0x7f8248049fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480dbfc0 0x7f8248049fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== osd.3 v2:172.21.15.81:6816/2629536107 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb00003ee50 con 0x7faff40777a0 2024-05-27T06:45:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824817a200 con 0x7f82480063c0 2024-05-27T06:45:10.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66e800640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824fe00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82480dbfc0 0x7f8248049fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:10.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== osd.2 v2:172.21.15.81:6808/2305164086 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76e0014ec0 con 0x7f76d40777d0 2024-05-27T06:45:10.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f824f400640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 0x7f82480d7c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f825007dc50 tx=0x7f825009af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 -- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6700b5bb0 msgr2=0x7fa670145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6700b5bb0 0x7fa670145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 -- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 msgr2=0x7fa67014e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa670156760 0x7fa67014e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6701591c0 con 0x7fa67006cb40 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66d400640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6700b5bb0 0x7fa670145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7fa66de00640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa670156f20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0190b0 tx=0x7fa65c019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f61ac018070 con 0x7f61c01567f0 2024-05-27T06:45:10.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250131040 con 0x7f82480063c0 2024-05-27T06:45:10.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825012cdf0 con 0x7f82480063c0 2024-05-27T06:45:10.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250135640 con 0x7f82480063c0 2024-05-27T06:45:10.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.606+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e24001040 con 0x7f1e500ce350 2024-05-27T06:45:10.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82480de100 con 0x7f82480063c0 2024-05-27T06:45:10.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82480de6c0 con 0x7f82480063c0 2024-05-27T06:45:10.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8248005580 con 0x7f82480063c0 2024-05-27T06:45:10.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e480145c0 con 0x7f1e500ce350 2024-05-27T06:45:10.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0bb020 con 0x7fa67006cb40 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c0b7680 con 0x7fa67006cb40 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0b2040 con 0x7fa67006cb40 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6700d1bb0 con 0x7fa67006cb40 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa670159710 con 0x7fa67006cb40 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e3006f600 0x7f1e30071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1e4808f930 con 0x7f1e500ce350 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] conn(0x7f1e300778d0 0x7f1e30079d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e4800df30 con 0x7f1e300778d0 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1e4808fd20 con 0x7f1e500ce350 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f1e54e00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] conn(0x7f1e300778d0 0x7f1e30079d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa670159af0 con 0x7fa67006cb40 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f61b6000640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f61a006f450 0x7f61a0071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.610+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f61ac08ec60 con 0x7f61c01567f0 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61bde00640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f61a006f450 0x7f61a0071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61b6000640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] conn(0x7f61a0077720 0x7f61a0079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61ac00fe50 con 0x7f61a0077720 2024-05-27T06:45:10.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f61ac0932a0 con 0x7f61c01567f0 2024-05-27T06:45:10.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f1e55800640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e3006f600 0x7f1e30071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61be800640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] conn(0x7f61a0077720 0x7f61a0079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f61bde00640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f61a006f450 0x7f61a0071960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61b80012f0 tx=0x7f61b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f1e54e00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] conn(0x7f1e300778d0 0x7f1e30079d20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f825012c4c0 con 0x7f82480063c0 2024-05-27T06:45:10.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7f1e55800640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e3006f600 0x7f1e30071b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e44000f70 tx=0x7f1e44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.614+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa65c0be020 con 0x7fa67006cb40 2024-05-27T06:45:10.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== osd.5 v2:172.21.15.106:6801/4003783298 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e4800df30 con 0x7f1e300778d0 2024-05-27T06:45:10.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa666000640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa65006f500 0x7fa650071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa65c128d10 con 0x7fa67006cb40 2024-05-27T06:45:10.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa666000640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] conn(0x7fa6500777d0 0x7fa650079c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f61be800640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] conn(0x7f61a0077720 0x7f61a0079b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa66d400640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa65006f500 0x7fa650071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa65c0abe50 con 0x7fa6500777d0 2024-05-27T06:45:10.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa65c12b370 con 0x7fa67006cb40 2024-05-27T06:45:10.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa66e800640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] conn(0x7fa6500777d0 0x7fa650079c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f8243400640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f823006f480 0x7f8230071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f82501a9340 con 0x7f82480063c0 2024-05-27T06:45:10.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa66d400640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa65006f500 0x7fa650071a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6700749c0 tx=0x7fa660009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7fa66e800640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] conn(0x7fa6500777d0 0x7fa650079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.622+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== osd.4 v2:172.21.15.106:6800/735979092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa65c0abe50 con 0x7fa6500777d0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f8243400640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] conn(0x7f8230077750 0x7f8230079ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f824ea00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f823006f480 0x7f8230071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8250072f30 con 0x7f8230077750 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f82501b0050 con 0x7f82480063c0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.618+0000 7f824fe00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] conn(0x7f8230077750 0x7f8230079ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.622+0000 7f824fe00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] conn(0x7f8230077750 0x7f8230079ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.622+0000 7f824ea00640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f823006f480 0x7f8230071990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f823c00b240 tx=0x7f823c00b3a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.622+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== osd.7 v2:172.21.15.106:6816/1738256587 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61ac00fe50 con 0x7f61a0077720 2024-05-27T06:45:10.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.622+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== osd.6 v2:172.21.15.106:6824/4186492770 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8250072f30 con 0x7f8230077750 2024-05-27T06:45:10.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.634+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 --> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f775c002de0 con 0x7f77680777d0 2024-05-27T06:45:10.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.634+0000 7f7784c00640 1 -- 172.21.15.81:0/3549272926 <== osd.0 v2:172.21.15.81:6822/1838218899 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f775c002de0 con 0x7f77680777d0 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.634+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] conn(0x7f77680777d0 msgr2=0x7f7768079c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.634+0000 7f777a000640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6822/1838218899,v1:172.21.15.81:6824/1838218899] conn(0x7f77680777d0 0x7f7768079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f776806f500 msgr2=0x7f7768071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f776806f500 0x7f7768071a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7780003ed0 tx=0x7f777c009040 comp rx=0 tx=0).stop 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 msgr2=0x7f77800e5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 --2- 172.21.15.81:0/3549272926 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f77800d9a30 0x7f77800e5cb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f77740b1040 tx=0x7f7770005840 comp rx=0 tx=0).stop 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f778dc00640 1 -- 172.21.15.81:0/3549272926 reap_dead start 2024-05-27T06:45:10.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.634+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 --> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f76d0002de0 con 0x7f76d40777d0 2024-05-27T06:45:10.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 --> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb014153ae0 con 0x7faff40777a0 2024-05-27T06:45:10.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76f0c00640 1 -- 172.21.15.81:0/1518898674 <== osd.2 v2:172.21.15.81:6808/2305164086 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f76d0002de0 con 0x7f76d40777d0 2024-05-27T06:45:10.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 shutdown_connections 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 --> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f61ac0c7d70 con 0x7f6188077650 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] conn(0x7f76d40777d0 msgr2=0x7f76d4079c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6808/2305164086,v1:172.21.15.81:6809/2305164086] conn(0x7f76d40777d0 0x7f76d4079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f76d406f500 msgr2=0x7f76d4071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f76d406f500 0x7f76d4071a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f76f40522c0 tx=0x7f76f406c380 comp rx=0 tx=0).stop 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec0dfdf0 msgr2=0x7f76ec0ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 --2- 172.21.15.81:0/1518898674 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76ec0dfdf0 0x7f76ec0ca0c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76ec04a2b0 tx=0x7f76e000c820 comp rx=0 tx=0).stop 2024-05-27T06:45:10.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76f9600640 1 -- 172.21.15.81:0/1518898674 reap_dead start 2024-05-27T06:45:10.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 >> 172.21.15.81:0/3549272926 conn(0x7f77800503c0 msgr2=0x7f7780046250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61a3400640 1 -- 172.21.15.81:0/3369081120 <== osd.1 v2:172.21.15.81:6800/404979354 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f61ac0c7d70 con 0x7f6188077650 2024-05-27T06:45:10.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 shutdown_connections 2024-05-27T06:45:10.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 >> 172.21.15.81:0/1518898674 conn(0x7f76ec00e390 msgr2=0x7f76ec006fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] conn(0x7f6188077650 msgr2=0x7f6188079aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6800/404979354,v1:172.21.15.81:6801/404979354] conn(0x7f6188077650 0x7f6188079aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 shutdown_connections 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 shutdown_connections 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f618806f400 msgr2=0x7f6188071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f618806f400 0x7f6188071910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f619c000f50 tx=0x7f619c006ef0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 msgr2=0x7f61ac1504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.638+0000 7f61b0e00640 1 --2- 172.21.15.81:0/3369081120 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61ac1e39f0 0x7f61ac1504f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009000 tx=0x7f6190005840 comp rx=0 tx=0).stop 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f61abe00640 1 -- 172.21.15.81:0/3369081120 reap_dead start 2024-05-27T06:45:10.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f76fb600640 1 -- 172.21.15.81:0/1518898674 wait complete. 2024-05-27T06:45:10.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 shutdown_connections 2024-05-27T06:45:10.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 >> 172.21.15.81:0/3369081120 conn(0x7f61ac06d150 msgr2=0x7f61ac06f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f777a000640 1 -- 172.21.15.81:0/3549272926 wait complete. 2024-05-27T06:45:10.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb00aa00640 1 -- 172.21.15.81:0/3861438388 <== osd.3 v2:172.21.15.81:6816/2629536107 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb014153ae0 con 0x7faff40777a0 2024-05-27T06:45:10.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 shutdown_connections 2024-05-27T06:45:10.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7f61b0e00640 1 -- 172.21.15.81:0/3369081120 wait complete. 2024-05-27T06:45:10.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] conn(0x7faff40777a0 msgr2=0x7faff4079bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6816/2629536107,v1:172.21.15.81:6817/2629536107] conn(0x7faff40777a0 0x7faff4079bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faff406f4d0 msgr2=0x7faff40719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faff406f4d0 0x7faff40719e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb004007fd0 tx=0x7fb004009040 comp rx=0 tx=0).stop 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0140c9d90 msgr2=0x7fb0140ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 --2- 172.21.15.81:0/3861438388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0140c9d90 0x7fb0140ccf00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0140c79c0 tx=0x7fb00000ecd0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb013400640 1 -- 172.21.15.81:0/3861438388 reap_dead start 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 shutdown_connections 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 >> 172.21.15.81:0/3861438388 conn(0x7fb01406cd90 msgr2=0x7fb01416a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 shutdown_connections 2024-05-27T06:45:10.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.642+0000 7fb019800640 1 -- 172.21.15.81:0/3861438388 wait complete. 2024-05-27T06:45:10.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.646+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 --> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e24002de0 con 0x7f1e300778d0 2024-05-27T06:45:10.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e4d600640 1 -- 172.21.15.81:0/316338654 <== osd.5 v2:172.21.15.106:6801/4003783298 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1e24002de0 con 0x7f1e300778d0 2024-05-27T06:45:10.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 --> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa67006a560 con 0x7fa6500777d0 2024-05-27T06:45:10.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] conn(0x7f1e300778d0 msgr2=0x7f1e30079d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:6801/4003783298,v1:172.21.15.106:6803/4003783298] conn(0x7f1e300778d0 0x7f1e30079d20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7fa666000640 1 -- 172.21.15.81:0/670172851 <== osd.4 v2:172.21.15.106:6800/735979092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa67006a560 con 0x7fa6500777d0 2024-05-27T06:45:10.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e3006f600 msgr2=0x7f1e30071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e3006f600 0x7f1e30071b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e44000f70 tx=0x7f1e44009040 comp rx=0 tx=0).stop 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 msgr2=0x7f1e5014cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.650+0000 7f1e42a00640 1 --2- 172.21.15.81:0/316338654 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e500ce350 0x7f1e5014cf10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1e48007d40 tx=0x7f1e4800e4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f1e56200640 1 -- 172.21.15.81:0/316338654 reap_dead start 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 shutdown_connections 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 >> 172.21.15.81:0/316338654 conn(0x7f1e5006cd90 msgr2=0x7f1e500c6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 shutdown_connections 2024-05-27T06:45:10.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f1e42a00640 1 -- 172.21.15.81:0/316338654 wait complete. 2024-05-27T06:45:10.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] conn(0x7fa6500777d0 msgr2=0x7fa650079c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:6800/735979092,v1:172.21.15.106:6802/735979092] conn(0x7fa6500777d0 0x7fa650079c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 --> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f61c0050870 con 0x7f61a0077720 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa65006f500 msgr2=0x7fa650071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa65006f500 0x7fa650071a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6700749c0 tx=0x7fa660009190 comp rx=0 tx=0).stop 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 msgr2=0x7fa670156f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 --2- 172.21.15.81:0/670172851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa67006cb40 0x7fa670156f20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0190b0 tx=0x7fa65c019240 comp rx=0 tx=0).stop 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa66e800640 1 -- 172.21.15.81:0/670172851 reap_dead start 2024-05-27T06:45:10.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7f61b6000640 1 -- 172.21.15.81:0/1433509198 <== osd.7 v2:172.21.15.106:6816/1738256587 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f61c0050870 con 0x7f61a0077720 2024-05-27T06:45:10.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 shutdown_connections 2024-05-27T06:45:10.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.654+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 >> 172.21.15.81:0/670172851 conn(0x7fa6700cecc0 msgr2=0x7fa670154c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 shutdown_connections 2024-05-27T06:45:10.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7fa675000640 1 -- 172.21.15.81:0/670172851 wait complete. 2024-05-27T06:45:10.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] conn(0x7f61a0077720 msgr2=0x7f61a0079b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:6816/1738256587,v1:172.21.15.106:6817/1738256587] conn(0x7f61a0077720 0x7f61a0079b70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f61a006f450 msgr2=0x7f61a0071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f61a006f450 0x7f61a0071960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61b80012f0 tx=0x7f61b8009040 comp rx=0 tx=0).stop 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 msgr2=0x7f61c015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 --2- 172.21.15.81:0/1433509198 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61c01567f0 0x7f61c015ef60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00b410 tx=0x7f61ac00b8d0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61be800640 1 -- 172.21.15.81:0/1433509198 reap_dead start 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 shutdown_connections 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 >> 172.21.15.81:0/1433509198 conn(0x7f61c006e790 msgr2=0x7f61c0152990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 shutdown_connections 2024-05-27T06:45:10.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f61ab400640 1 -- 172.21.15.81:0/1433509198 wait complete. 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 --> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f824800e710 con 0x7f8230077750 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.658+0000 7f8243400640 1 -- 172.21.15.81:0/961733147 <== osd.6 v2:172.21.15.106:6824/4186492770 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f824800e710 con 0x7f8230077750 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] conn(0x7f8230077750 msgr2=0x7f8230079ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:6824/4186492770,v1:172.21.15.106:6825/4186492770] conn(0x7f8230077750 0x7f8230079ba0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f823006f480 msgr2=0x7f8230071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f823006f480 0x7f8230071990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f823c00b240 tx=0x7f823c00b3a0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 msgr2=0x7f82480d7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 --2- 172.21.15.81:0/961733147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82480063c0 0x7f82480d7c60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f825007dc50 tx=0x7f825009af90 comp rx=0 tx=0).stop 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f824fe00640 1 -- 172.21.15.81:0/961733147 reap_dead start 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 shutdown_connections 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 >> 172.21.15.81:0/961733147 conn(0x7f82480503c0 msgr2=0x7f82480d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 shutdown_connections 2024-05-27T06:45:10.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.662+0000 7f8255000640 1 -- 172.21.15.81:0/961733147 wait complete. 2024-05-27T06:45:10.677 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:10.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-27T06:45:10.679 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:10.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-27T06:45:10.683 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:10.684 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-27T06:45:10.689 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:10.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-27T06:45:10.696 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:10.696 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:10.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-27T06:45:10.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-27T06:45:10.698 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:10.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-27T06:45:10.701 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:10.701 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-27T06:45:10.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fe367200640 1 Processor -- start 2024-05-27T06:45:10.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fe367200640 1 -- start start 2024-05-27T06:45:10.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fda23000640 1 Processor -- start 2024-05-27T06:45:10.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fda23000640 1 -- start start 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fe367200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3600487e0 0x7fe36004eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fe367200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 0x7fe36004d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.942+0000 7fe365e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3600487e0 0x7fe36004eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe367200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36004f8d0 con 0x7fe36004d360 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe367200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe3600e1500 con 0x7fe360047b70 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe367200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe3600e08f0 con 0x7fe3600487e0 2024-05-27T06:45:10.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe3600487e0 0x7fe36004eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46926/0 (socket says 172.21.15.81:46926) 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 -- 172.21.15.81:0/1972116806 learned_addr learned my addr 172.21.15.81:0/1972116806 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda23000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda23000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda1c072700 0x7fda1c072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda23000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1c158d60 con 0x7fda1c15c580 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda23000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fda1c15bb30 con 0x7fda1c072700 2024-05-27T06:45:10.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda23000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fda1c159040 con 0x7fda1c0c8b30 2024-05-27T06:45:10.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe366800640 1 --2- 172.21.15.81:0/1972116806 >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 0x7fe36004d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fe36004d360 0x7fe360047460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59130/0 (socket says 172.21.15.81:59130) 2024-05-27T06:45:10.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda21000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda1c072700 0x7fda1c072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46930/0 (socket says 172.21.15.81:46930) 2024-05-27T06:45:10.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 -- 172.21.15.81:0/1972116806 >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 msgr2=0x7fe36004d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 -- 172.21.15.81:0/4092002108 learned_addr learned my addr 172.21.15.81:0/4092002108 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 -- 172.21.15.81:0/4092002108 >> v2:172.21.15.81:3300/0 conn(0x7fda1c072700 msgr2=0x7fda1c072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 --2- 172.21.15.81:0/4092002108 >> v2:172.21.15.81:3300/0 conn(0x7fda1c072700 0x7fda1c072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 --2- 172.21.15.81:0/1972116806 >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 0x7fe36004d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 -- 172.21.15.81:0/1972116806 >> v1:172.21.15.106:6789/0 conn(0x7fe36004d360 legacy=0x7fe360047460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 -- 172.21.15.81:0/4092002108 >> v1:172.21.15.81:6789/0 conn(0x7fda1c15c580 legacy=0x7fda1c158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fda1be00640 1 -- 172.21.15.81:0/4092002108 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c159320 con 0x7fda1c0c8b30 2024-05-27T06:45:10.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe364a00640 1 -- 172.21.15.81:0/1972116806 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1832951054 0 0) 0x7fe36004f8d0 con 0x7fe36004d360 2024-05-27T06:45:10.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe365e00640 1 -- 172.21.15.81:0/1972116806 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3600e0bd0 con 0x7fe3600487e0 2024-05-27T06:45:10.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.946+0000 7fe366800640 1 --2- 172.21.15.81:0/1972116806 >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 0x7fe36004d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:10.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.950+0000 7fda1be00640 1 --2- 172.21.15.81:0/4092002108 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0cac80 tx=0x7fda1001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea0bf961c816bce4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.950+0000 7fda1aa00640 1 -- 172.21.15.81:0/4092002108 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100b5070 con 0x7fda1c0c8b30 2024-05-27T06:45:10.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.950+0000 7fda1aa00640 1 -- 172.21.15.81:0/4092002108 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda100a89f0 con 0x7fda1c0c8b30 2024-05-27T06:45:10.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.950+0000 7fda1aa00640 1 -- 172.21.15.81:0/4092002108 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100a8e00 con 0x7fda1c0c8b30 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 -- 172.21.15.81:0/4092002108 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 msgr2=0x7fda1c0ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 --2- 172.21.15.81:0/4092002108 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0cac80 tx=0x7fda1001ed30 comp rx=0 tx=0).stop 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 -- 172.21.15.81:0/4092002108 shutdown_connections 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 --2- 172.21.15.81:0/4092002108 >> v2:172.21.15.81:3300/0 conn(0x7fda1c072700 0x7fda1c072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 --2- 172.21.15.81:0/4092002108 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c0ca6a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 -- 172.21.15.81:0/4092002108 >> 172.21.15.81:0/4092002108 conn(0x7fda1c06fea0 msgr2=0x7fda1c0702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 -- 172.21.15.81:0/4092002108 shutdown_connections 2024-05-27T06:45:10.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 -- 172.21.15.81:0/4092002108 wait complete. 2024-05-27T06:45:10.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.954+0000 7fda23000640 1 Processor -- start 2024-05-27T06:45:10.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7f4ddc000640 1 Processor -- start 2024-05-27T06:45:10.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 -- start start 2024-05-27T06:45:10.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 0x7fda1c14c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 0x7fda1c158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c15c990 con 0x7fda1c072700 2024-05-27T06:45:10.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fda1c15c060 con 0x7fda1c0c8b30 2024-05-27T06:45:10.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda23000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fda1c0c8020 con 0x7fda1c15c580 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda1be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 0x7fda1c14c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda1be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 0x7fda1c14c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57362/0 (socket says 172.21.15.81:57362) 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda1be00640 1 -- 172.21.15.81:0/1941358123 learned_addr learned my addr 172.21.15.81:0/1941358123 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 0x7fda1c158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7f4ddc000640 1 -- start start 2024-05-27T06:45:10.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7f4ddc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 -- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 msgr2=0x7fda1c1580c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c1580c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 -- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 msgr2=0x7fda1c14c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 0x7fda1c14c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 -- 172.21.15.81:0/1941358123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1c156dd0 con 0x7fda1c15c580 2024-05-27T06:45:10.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7f4dd9600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.958+0000 7fda21000640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 0x7fda1c158600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda14007d20 tx=0x7fda1400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1400edd0 con 0x7fda1c15c580 2024-05-27T06:45:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54934/0 (socket says 172.21.15.81:54934) 2024-05-27T06:45:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 -- 172.21.15.81:0/1869657172 learned_addr learned my addr 172.21.15.81:0/1869657172 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fe365e00640 1 --2- 172.21.15.81:0/1972116806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3600487e0 0x7fe36004eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe36004d080 tx=0x7fe35401ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a5f2cf4b38675a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda140039d0 con 0x7fda1c15c580 2024-05-27T06:45:10.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14003dc0 con 0x7fda1c15c580 2024-05-27T06:45:10.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4ddc000640 1 --2- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 0x7f4dd41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fe364a00640 1 -- 172.21.15.81:0/1972116806 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540b5070 con 0x7fe3600487e0 2024-05-27T06:45:10.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fe364a00640 1 -- 172.21.15.81:0/1972116806 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3540a8a10 con 0x7fe3600487e0 2024-05-27T06:45:10.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fe364a00640 1 -- 172.21.15.81:0/1972116806 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540a8de0 con 0x7fe3600487e0 2024-05-27T06:45:10.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dd4161b20 con 0x7f4dd4073ef0 2024-05-27T06:45:10.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4dd40cb260 con 0x7f4dd41611d0 2024-05-27T06:45:10.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd40ca6e0 con 0x7f4dd40735c0 2024-05-27T06:45:10.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd8c00640 1 --2- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 0x7f4dd41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda23000640 1 -- 172.21.15.81:0/1941358123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1c1570b0 con 0x7fda1c15c580 2024-05-27T06:45:10.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 -- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 msgr2=0x7f4dd41615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 --2- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 0x7f4dd41615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 -- 172.21.15.81:0/1869657172 >> v1:172.21.15.81:6789/0 conn(0x7f4dd4073ef0 legacy=0x7f4dd4074300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda23000640 1 -- 172.21.15.81:0/1941358123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda1c14e280 con 0x7fda1c15c580 2024-05-27T06:45:10.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd9600640 1 -- 172.21.15.81:0/1869657172 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd40ca9c0 con 0x7f4dd40735c0 2024-05-27T06:45:10.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f00052a0 con 0x7fda1c15c580 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.962+0000 7f4dd8c00640 1 --2- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 0x7f4dd41615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 -- 172.21.15.81:0/1972116806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3600487e0 msgr2=0x7fe36004eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 --2- 172.21.15.81:0/1972116806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3600487e0 0x7fe36004eff0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe36004d080 tx=0x7fe35401ed10 comp rx=0 tx=0).stop 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 -- 172.21.15.81:0/1972116806 shutdown_connections 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 --2- 172.21.15.81:0/1972116806 >> v2:172.21.15.81:3300/0 conn(0x7fe360047b70 0x7fe36004d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 --2- 172.21.15.81:0/1972116806 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3600487e0 0x7fe36004eff0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fe367200640 1 -- 172.21.15.81:0/1972116806 >> 172.21.15.81:0/1972116806 conn(0x7fe36000a000 msgr2=0x7fe36000a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7faf87e00640 1 Processor -- start 2024-05-27T06:45:10.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7faf87e00640 1 -- start start 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda1401b020 con 0x7fda1c15c580 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fda0fe00640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 0x7fd9fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fda1408f7f0 con 0x7fda1c15c580 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fda1be00640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 0x7fd9fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.966+0000 7fda1be00640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 0x7fd9fc071940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda1001ed70 tx=0x7fda10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7ff444000640 1 Processor -- start 2024-05-27T06:45:10.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fe367200640 1 -- 172.21.15.81:0/1972116806 shutdown_connections 2024-05-27T06:45:10.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fe367200640 1 -- 172.21.15.81:0/1972116806 wait complete. 2024-05-27T06:45:10.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fa7a0800640 1 Processor -- start 2024-05-27T06:45:10.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fa7a0800640 1 -- start start 2024-05-27T06:45:10.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fda1405c750 con 0x7fda1c15c580 2024-05-27T06:45:10.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7f4dd9600640 1 --2- 172.21.15.81:0/1869657172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009e50 tx=0x7f4dcc01f230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9d94b214a455c8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7ff444000640 1 -- start start 2024-05-27T06:45:10.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7faf87e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faf8004d9a0 0x7faf80007dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7faf87e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf8004ce40 con 0x7faf80046470 2024-05-27T06:45:10.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7faf87e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf800dc070 con 0x7faf800d64d0 2024-05-27T06:45:10.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7faf87e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7faf8004d0e0 con 0x7faf8004d9a0 2024-05-27T06:45:10.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.970+0000 7fa7a0800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7980cab10 0x7fa7980cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf87400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7faf800d64d0 0x7faf800e6930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59142/0 (socket says 172.21.15.81:59142) 2024-05-27T06:45:10.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf87400640 1 -- 172.21.15.81:0/2410045783 learned_addr learned my addr 172.21.15.81:0/2410045783 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff444000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff43c073ef0 0x7ff43c074300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff441600640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff43c0735c0 0x7ff43c0c92f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59150/0 (socket says 172.21.15.81:59150) 2024-05-27T06:45:10.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa7a0800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7980caf20 0x7fa7980ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa7a0800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7980c9790 con 0x7fa798164190 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa7a0800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa7980c9a70 con 0x7fa7980cab10 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa7a0800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa79806d1a0 con 0x7fa7980caf20 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf86000640 1 --2- 172.21.15.81:0/2410045783 >> v2:172.21.15.106:3300/0 conn(0x7faf8004d9a0 0x7faf80007dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4dd3e00640 1 -- 172.21.15.81:0/1869657172 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0b4070 con 0x7f4dd40735c0 2024-05-27T06:45:10.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4dd3e00640 1 -- 172.21.15.81:0/1869657172 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0af400 con 0x7f4dd40735c0 2024-05-27T06:45:10.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4dd3e00640 1 -- 172.21.15.81:0/1869657172 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0af7d0 con 0x7f4dd40735c0 2024-05-27T06:45:10.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7980caf20 0x7fa7980ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff441600640 1 -- 172.21.15.81:0/490835866 learned_addr learned my addr 172.21.15.81:0/490835866 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7980caf20 0x7fa7980ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46966/0 (socket says 172.21.15.81:46966) 2024-05-27T06:45:10.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 -- 172.21.15.81:0/2236973561 learned_addr learned my addr 172.21.15.81:0/2236973561 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728199771 0 0) 0x7faf800dc070 con 0x7faf800d64d0 2024-05-27T06:45:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf6c003530 con 0x7faf800d64d0 2024-05-27T06:45:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79de00640 1 --2- 172.21.15.81:0/2236973561 >> v2:172.21.15.81:3300/0 conn(0x7fa7980cab10 0x7fa7980cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff442000640 1 --2- 172.21.15.81:0/490835866 >> v2:172.21.15.81:3300/0 conn(0x7ff43c073ef0 0x7ff43c074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43c0ca6e0 con 0x7ff43c1611d0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff43c0ca9c0 con 0x7ff43c0735c0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff43c0cb260 con 0x7ff43c073ef0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 390109603 0 0) 0x7faf6c003530 con 0x7faf800d64d0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 >> v2:172.21.15.106:3300/0 conn(0x7faf8004d9a0 msgr2=0x7faf80007dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 --2- 172.21.15.81:0/2410045783 >> v2:172.21.15.106:3300/0 conn(0x7faf8004d9a0 0x7faf80007dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 >> v1:172.21.15.81:6789/0 conn(0x7faf80046470 legacy=0x7faf80046d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 --> v1:172.21.15.106:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8004d3c0 con 0x7faf800d64d0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.1 v1:172.21.15.106:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7faf88078030 con 0x7faf800d64d0 2024-05-27T06:45:10.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 >> v1:172.21.15.106:6789/0 conn(0x7faf800d64d0 legacy=0x7faf800e6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff43be00640 1 -- 172.21.15.81:0/490835866 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197372306 0 0) 0x7ff43c0ca9c0 con 0x7ff43c0735c0 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff43be00640 1 -- 172.21.15.81:0/490835866 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff424003530 con 0x7ff43c0735c0 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fe367200640 1 Processor -- start 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf6c0039c0 0x7faf6c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6c004f50 0x7faf6c00ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf8004d0e0 con 0x7faf6c004f50 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faf6c010520 con 0x7faf6c0039c0 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7faf6c0106a0 con 0x7faf6c004370 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 -- 172.21.15.81:0/2236973561 >> v2:172.21.15.81:3300/0 conn(0x7fa7980cab10 msgr2=0x7fa7980cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 --2- 172.21.15.81:0/2236973561 >> v2:172.21.15.81:3300/0 conn(0x7fa7980cab10 0x7fa7980cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 -- 172.21.15.81:0/2236973561 >> v1:172.21.15.81:6789/0 conn(0x7fa798164190 legacy=0x7fa7981645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fa79d400640 1 -- 172.21.15.81:0/2236973561 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79806d480 con 0x7fa7980caf20 2024-05-27T06:45:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7fe367200640 1 -- start start 2024-05-27T06:45:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff442000640 1 -- 172.21.15.81:0/490835866 >> v1:172.21.15.106:6789/0 conn(0x7ff43c0735c0 legacy=0x7ff43c0c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff442000640 1 -- 172.21.15.81:0/490835866 >> v1:172.21.15.81:6789/0 conn(0x7ff43c1611d0 legacy=0x7ff43c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7ff442000640 1 -- 172.21.15.81:0/490835866 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c0caca0 con 0x7ff43c073ef0 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 msgr2=0x7f4dd40c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 --2- 172.21.15.81:0/1869657172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009e50 tx=0x7f4dcc01f230 comp rx=0 tx=0).stop 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 shutdown_connections 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 --2- 172.21.15.81:0/1869657172 >> v2:172.21.15.106:3300/0 conn(0x7f4dd41611d0 0x7f4dd41615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 --2- 172.21.15.81:0/1869657172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd40735c0 0x7f4dd40c92f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.974+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 >> 172.21.15.81:0/1869657172 conn(0x7f4dd406dca0 msgr2=0x7f4dd406e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86a00640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf6c0039c0 0x7faf6c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 0x7fe3600cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3600487e0 0x7fe360052740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe365e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 0x7fe3600cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 0x7fe3600c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe365e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 0x7fe3600cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54952/0 (socket says 172.21.15.81:54952) 2024-05-27T06:45:10.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe365e00640 1 -- 172.21.15.81:0/2346467756 learned_addr learned my addr 172.21.15.81:0/2346467756 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3600de100 con 0x7fe3600487e0 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe3600e1190 con 0x7fe36004d360 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe3600de280 con 0x7fe360047b70 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 0x7fe3600c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe365400640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3600487e0 0x7fe360052740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 -- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 msgr2=0x7fe3600cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 0x7fe3600cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 -- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3600487e0 msgr2=0x7fe360052740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3600487e0 0x7fe360052740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fe366800640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3600daba0 con 0x7fe36004d360 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 -- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf6c0039c0 msgr2=0x7faf6c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf6c0039c0 0x7faf6c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 -- 172.21.15.81:0/2410045783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6c004f50 msgr2=0x7faf6c00ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf6c004f50 0x7faf6c00ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86000640 1 -- 172.21.15.81:0/2410045783 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf8004d3c0 con 0x7faf6c004370 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf87400640 1 -- 172.21.15.81:0/2410045783 reap_dead start 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7faf86a00640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf6c0039c0 0x7faf6c003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 shutdown_connections 2024-05-27T06:45:10.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7f4ddc000640 1 -- 172.21.15.81:0/1869657172 wait complete. 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fa79d400640 1 --2- 172.21.15.81:0/2236973561 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980caf20 0x7fa7980ccc90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7980704f0 tx=0x7fa78801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b6a87ebf05b792e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fa79ca00640 1 -- 172.21.15.81:0/2236973561 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880b5070 con 0x7fa7980caf20 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fa79ca00640 1 -- 172.21.15.81:0/2236973561 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7880a8a10 con 0x7fa7980caf20 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.978+0000 7fa79ca00640 1 -- 172.21.15.81:0/2236973561 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7880b5070 con 0x7fa7980caf20 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe366800640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 0x7fe3600c7e40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe35800d5a0 tx=0x7fe35800da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff442000640 1 --2- 172.21.15.81:0/490835866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c073ef0 0x7ff43c074300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0cb3e0 tx=0x7ff43401efc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab13beae8e7808 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f1f55800640 1 Processor -- start 2024-05-27T06:45:10.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 Processor -- start 2024-05-27T06:45:10.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 -- 172.21.15.81:0/2236973561 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980caf20 msgr2=0x7fa7980ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 --2- 172.21.15.81:0/2236973561 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980caf20 0x7fa7980ccc90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7980704f0 tx=0x7fa78801ed30 comp rx=0 tx=0).stop 2024-05-27T06:45:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff43be00640 1 -- 172.21.15.81:0/490835866 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4340b5070 con 0x7ff43c073ef0 2024-05-27T06:45:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 -- 172.21.15.81:0/2236973561 shutdown_connections 2024-05-27T06:45:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 --2- 172.21.15.81:0/2236973561 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980caf20 0x7fa7980ccc90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 --2- 172.21.15.81:0/2236973561 >> v2:172.21.15.81:3300/0 conn(0x7fa7980cab10 0x7fa7980cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 -- start start 2024-05-27T06:45:10.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 -- 172.21.15.81:0/2236973561 >> 172.21.15.81:0/2236973561 conn(0x7fa79806f080 msgr2=0x7fa79806f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358018020 con 0x7fe36004d360 2024-05-27T06:45:10.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff43be00640 1 -- 172.21.15.81:0/490835866 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4340a8b90 con 0x7ff43c073ef0 2024-05-27T06:45:10.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 -- 172.21.15.81:0/2236973561 shutdown_connections 2024-05-27T06:45:10.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff43be00640 1 -- 172.21.15.81:0/490835866 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430001700 con 0x7ff43c073ef0 2024-05-27T06:45:10.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fa7a0800640 1 -- 172.21.15.81:0/2236973561 wait complete. 2024-05-27T06:45:10.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c073ef0 msgr2=0x7ff43c074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff444000640 1 --2- 172.21.15.81:0/490835866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c073ef0 0x7ff43c074300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0cb3e0 tx=0x7ff43401efc0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 shutdown_connections 2024-05-27T06:45:10.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3600cd6c0 con 0x7fe36004d360 2024-05-27T06:45:10.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe358014620 con 0x7fe36004d360 2024-05-27T06:45:10.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff444000640 1 --2- 172.21.15.81:0/490835866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c073ef0 0x7ff43c074300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 >> 172.21.15.81:0/490835866 conn(0x7ff43c06dca0 msgr2=0x7ff43c06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580149f0 con 0x7fe36004d360 2024-05-27T06:45:10.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 0x7f4dd4155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7faf86000640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf6c004810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf74009ae0 tx=0x7faf7401ca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb02bd9515f553a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd4073ef0 0x7f4dd4153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 0x7f4dd40c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd4160cb0 con 0x7f4dd41611d0 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd40caf80 con 0x7f4dd40735c0 2024-05-27T06:45:10.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4ddc000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd40c9830 con 0x7f4dd4073ef0 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3600cc6a0 con 0x7fe36004d360 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4dd9600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 0x7f4dd4155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd4073ef0 0x7f4dd4153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4dd9600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 0x7f4dd4155590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46988/0 (socket says 172.21.15.81:46988) 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4dd9600640 1 -- 172.21.15.81:0/2860671588 learned_addr learned my addr 172.21.15.81:0/2860671588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f4dda000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 0x7f4dd40c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf740b2070 con 0x7faf6c004370 2024-05-27T06:45:10.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf740a5660 con 0x7faf6c004370 2024-05-27T06:45:10.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7faf85600640 1 -- 172.21.15.81:0/2410045783 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf740096b0 con 0x7faf6c004370 2024-05-27T06:45:10.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.982+0000 7f1f55800640 1 -- start start 2024-05-27T06:45:10.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 -- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 msgr2=0x7faf6c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf6c004810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faf74009ae0 tx=0x7faf7401ca90 comp rx=0 tx=0).stop 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 -- 172.21.15.81:0/2410045783 shutdown_connections 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 --2- 172.21.15.81:0/2410045783 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf6c004810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 -- 172.21.15.81:0/2410045783 >> 172.21.15.81:0/2410045783 conn(0x7faf800514d0 msgr2=0x7faf800518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 -- 172.21.15.81:0/2410045783 shutdown_connections 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 -- 172.21.15.81:0/2410045783 wait complete. 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 -- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd4073ef0 msgr2=0x7f4dd4153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd4073ef0 0x7f4dd4153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 -- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 msgr2=0x7f4dd4155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 0x7f4dd4155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 -- 172.21.15.81:0/2860671588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd4154230 con 0x7f4dd41611d0 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dda000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 0x7f4dd40c6ad0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4075770 tx=0x7f4dc800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8017070 con 0x7f4dd41611d0 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7fa7a0800640 1 Processor -- start 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7fa7a0800640 1 -- start start 2024-05-27T06:45:10.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f55800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1f480457e0 0x7f1f4804b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f55800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f480e7960 con 0x7f1f480dda20 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f55800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f480e4620 con 0x7f1f4804bbd0 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f55800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1f480dde30 con 0x7f1f480457e0 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4fe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1f480dda20 0x7f1f480060a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37110/0 (socket says 172.21.15.81:37110) 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4fe00640 1 -- 172.21.15.81:0/3104864536 learned_addr learned my addr 172.21.15.81:0/3104864536 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4f400640 1 --2- 172.21.15.81:0/3104864536 >> v2:172.21.15.106:3300/0 conn(0x7f1f480457e0 0x7f1f4804b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2965913662 0 0) 0x7f1f480e7960 con 0x7f1f480dda20 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f38003530 con 0x7f1f480dda20 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3945212703 0 0) 0x7f1f38003530 con 0x7f1f480dda20 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 >> v2:172.21.15.106:3300/0 conn(0x7f1f480457e0 msgr2=0x7f1f4804b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 --2- 172.21.15.81:0/3104864536 >> v2:172.21.15.106:3300/0 conn(0x7f1f480457e0 0x7f1f4804b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 >> v1:172.21.15.106:6789/0 conn(0x7f1f4804bbd0 legacy=0x7f1f480ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:10.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f480e3860 con 0x7f1f480dda20 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f1f3c001700 con 0x7f1f480dda20 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 >> v1:172.21.15.81:6789/0 conn(0x7f1f480dda20 legacy=0x7f1f480060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 0x7f1f38003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f38004810 0x7f1f38004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38005280 0x7f1f3800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1f480dde30 con 0x7f1f38005280 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3800c1f0 con 0x7f1f38004810 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3800c3f0 con 0x7f1f38003a10 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 0x7f1f38003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 shutdown_connections 2024-05-27T06:45:10.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7ff444000640 1 -- 172.21.15.81:0/490835866 wait complete. 2024-05-27T06:45:10.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.986+0000 7faf87e00640 1 Processor -- start 2024-05-27T06:45:10.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc80052d0 con 0x7f4dd41611d0 2024-05-27T06:45:10.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd41f7710 con 0x7f4dd41611d0 2024-05-27T06:45:10.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f1f4fe00640 1 -- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f38004810 msgr2=0x7f1f38004c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T06:45:10.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8003820 con 0x7f4dd41611d0 2024-05-27T06:45:10.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 Processor -- start 2024-05-27T06:45:10.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3600059d0 con 0x7fe36004d360 2024-05-27T06:45:10.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 -- start start 2024-05-27T06:45:10.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd41f7cd0 con 0x7f4dd41611d0 2024-05-27T06:45:10.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe35801b020 con 0x7fe36004d360 2024-05-27T06:45:10.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fe35e000640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 0x7fe3480719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe35808ed00 con 0x7fe36004d360 2024-05-27T06:45:10.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd406a560 con 0x7f4dd41611d0 2024-05-27T06:45:10.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fa7a0800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 -- start start 2024-05-27T06:45:10.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 0x7ff43c1f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fa7a0800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7980caf20 0x7fa79814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fa7a0800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa798164190 0x7fa79814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fa79de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7fa79de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47014/0 (socket says 172.21.15.81:47014) 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 0x7ff43c0c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff441600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 0x7ff43c1f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff441600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 0x7ff43c1f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47012/0 (socket says 172.21.15.81:47012) 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff440c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff440c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54982/0 (socket says 172.21.15.81:54982) 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff441600640 1 -- 172.21.15.81:0/252876794 learned_addr learned my addr 172.21.15.81:0/252876794 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c074840 con 0x7ff43c1611d0 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c160cb0 con 0x7ff43c0735c0 2024-05-27T06:45:10.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff43c15e4c0 con 0x7ff43c073ef0 2024-05-27T06:45:11.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fe365e00640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 0x7fe3480719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 -- 172.21.15.81:0/1551989235 learned_addr learned my addr 172.21.15.81:0/1551989235 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:11.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7980cdf90 con 0x7fa7980caf20 2024-05-27T06:45:11.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa79806da40 con 0x7fa7980cab10 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa7980c93c0 con 0x7fa798164190 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79d400640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7980caf20 0x7fa79814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f38004810 0x7f1f38004c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f1f4fe00640 1 -- 172.21.15.81:0/3104864536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38005280 msgr2=0x7f1f3800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7f1f54e00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38005280 0x7f1f3800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 -- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 msgr2=0x7ff43c1f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 0x7ff43c1f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff442000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 0x7ff43c0c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 -- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 msgr2=0x7ff43c0c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 0x7ff43c0c6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c1f7930 con 0x7ff43c073ef0 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf800d69e0 0x7faf800d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf80175950 0x7faf800d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf8004c0e0 con 0x7faf80175950 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faf800dc610 con 0x7faf800d69e0 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf87e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7faf8004c260 con 0x7faf6c004370 2024-05-27T06:45:11.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf86a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf86a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:54990/0 (socket says 172.21.15.81:54990) 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.990+0000 7faf86a00640 1 -- 172.21.15.81:0/2317690801 learned_addr learned my addr 172.21.15.81:0/2317690801 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86000640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf800d69e0 0x7faf800d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf87400640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf80175950 0x7faf800d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 -- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf800d69e0 msgr2=0x7faf800d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf800d69e0 0x7faf800d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 -- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf80175950 msgr2=0x7faf800d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf80175950 0x7faf800d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 -- 172.21.15.81:0/2317690801 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf800dbe20 con 0x7faf6c004370 2024-05-27T06:45:11.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4dc801a070 con 0x7f4dd41611d0 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 -- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa798164190 msgr2=0x7fa79814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa798164190 0x7fa79814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 -- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7980caf20 msgr2=0x7fa79814d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7980caf20 0x7fa79814d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7981f7170 con 0x7fa7980cab10 2024-05-27T06:45:11.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff442000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 0x7ff43c0c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:11.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fa79de00640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00d620 tx=0x7fa78c00dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf86a00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf7800e780 tx=0x7faf7800ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7fe365e00640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 0x7fe3480719e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe354000c10 tx=0x7fe354008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78006b40 con 0x7faf6c004370 2024-05-27T06:45:11.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf78006ce0 con 0x7faf6c004370 2024-05-27T06:45:11.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78003bc0 con 0x7faf6c004370 2024-05-27T06:45:11.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf800da6c0 con 0x7faf6c004370 2024-05-27T06:45:11.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf80175150 con 0x7faf6c004370 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38005280 0x7f1f3800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f1f4fe00640 1 -- 172.21.15.81:0/3104864536 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f480e3860 con 0x7f1f38003a10 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f1f54e00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38005280 0x7f1f3800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f1f54e00640 1 -- 172.21.15.81:0/3104864536 reap_dead start 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f4dd1600640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 0x7f4db4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4dc808e420 con 0x7f4dd41611d0 2024-05-27T06:45:11.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7ff440c00640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c3c0 tx=0x7ff42800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.994+0000 7f4dd9600640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 0x7f4db4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428017070 con 0x7ff43c073ef0 2024-05-27T06:45:11.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4280052d0 con 0x7ff43c073ef0 2024-05-27T06:45:11.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43c155740 con 0x7ff43c073ef0 2024-05-27T06:45:11.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43c155ca0 con 0x7ff43c073ef0 2024-05-27T06:45:11.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff428003800 con 0x7ff43c073ef0 2024-05-27T06:45:11.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf800059d0 con 0x7faf6c004370 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 0x7f1f38003e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c320 tx=0x7f1f3c01ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ed0bc3518ce1a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c007da0 con 0x7f1f38003a10 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f3c0a9020 con 0x7f1f38003a10 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f4ea00640 1 -- 172.21.15.81:0/3104864536 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0a5c10 con 0x7f1f38003a10 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7faf78005070 con 0x7faf6c004370 2024-05-27T06:45:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f4dd9600640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 0x7f4db4071940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc000f90 tx=0x7f4dcc0aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4180052a0 con 0x7ff43c073ef0 2024-05-27T06:45:11.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f55800640 1 -- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 msgr2=0x7f1f38003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f55800640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 0x7f1f38003e60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c320 tx=0x7f1f3c01ccb0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00b620 con 0x7fa7980cab10 2024-05-27T06:45:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f55800640 1 -- 172.21.15.81:0/3104864536 shutdown_connections 2024-05-27T06:45:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f55800640 1 --2- 172.21.15.81:0/3104864536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f38003a10 0x7f1f38003e60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7981f7450 con 0x7fa7980cab10 2024-05-27T06:45:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7f1f55800640 1 -- 172.21.15.81:0/3104864536 >> 172.21.15.81:0/3104864536 conn(0x7f1f480503c0 msgr2=0x7f1f480507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa78c019070 con 0x7fa7980cab10 2024-05-27T06:45:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:10.998+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe35805bc70 con 0x7fe36004d360 2024-05-27T06:45:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7faf7ea00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 0x7faf68071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7981515c0 con 0x7fa7980cab10 2024-05-27T06:45:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00b9c0 con 0x7fa7980cab10 2024-05-27T06:45:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7f1f55800640 1 -- 172.21.15.81:0/3104864536 shutdown_connections 2024-05-27T06:45:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7f1f55800640 1 -- 172.21.15.81:0/3104864536 wait complete. 2024-05-27T06:45:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faf7808eef0 con 0x7faf6c004370 2024-05-27T06:45:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7faf86000640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 0x7faf68071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7faf86000640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 0x7faf68071960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faf74009ae0 tx=0x7faf74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4dc8021080 con 0x7f4dd41611d0 2024-05-27T06:45:11.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff42801a070 con 0x7ff43c073ef0 2024-05-27T06:45:11.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7f1f55800640 1 Processor -- start 2024-05-27T06:45:11.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7faf7805be50 con 0x7faf6c004370 2024-05-27T06:45:11.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7f1f55800640 1 -- start start 2024-05-27T06:45:11.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7740052a0 con 0x7fa7980cab10 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.002+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa78c00bb60 con 0x7fa7980cab10 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7fa792000640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 0x7fa77c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa78c08f620 con 0x7fa7980cab10 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7ff439600640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 0x7ff41c0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff42808e590 con 0x7ff43c073ef0 2024-05-27T06:45:11.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7fa79d400640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 0x7fa77c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7ff441600640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 0x7ff41c0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7f1f55800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7ff441600640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 0x7ff41c0719e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff430000f70 tx=0x7ff430001b10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7fa79d400640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 0x7fa77c071940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7880b2000 tx=0x7fa7880aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.006+0000 7f1f55800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 0x7f1f480e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57450/0 (socket says 172.21.15.81:57450) 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 -- 172.21.15.81:0/1684816462 learned_addr learned my addr 172.21.15.81:0/1684816462 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 0x7f1f480e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1f480e36c0 con 0x7f1f38003a10 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1f480e3e20 con 0x7f1f480059a0 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1f480e4100 con 0x7f1f48036250 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4f400640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 0x7f1f480e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f54e00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 0x7f1f480e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff428005480 con 0x7ff43c073ef0 2024-05-27T06:45:11.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 -- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 msgr2=0x7f1f480e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa78c05c500 con 0x7fa7980cab10 2024-05-27T06:45:11.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 0x7f1f480e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 -- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 msgr2=0x7f1f480e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 0x7f1f480e5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f480e4880 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f54e00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 0x7f1f480e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4f400640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 0x7f1f480e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f4fe00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c320 tx=0x7f1f38005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0016d0 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f3c0047f0 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.010+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c004c00 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.014+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f48035c40 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.014+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f48034c20 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.018+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1f3c0ae430 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.018+0000 7f1f43e00640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 0x7f1f300719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.018+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1f3c128830 con 0x7f1f38003a10 2024-05-27T06:45:11.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.018+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f2c0052a0 con 0x7f1f38003a10 2024-05-27T06:45:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.022+0000 7f1f4f400640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 0x7f1f300719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.022+0000 7f1f4f400640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 0x7f1f300719e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1f4804dc90 tx=0x7f1f44008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.030+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1f3c0b4070 con 0x7f1f38003a10 2024-05-27T06:45:11.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.034+0000 7fe9d3e00640 1 Processor -- start 2024-05-27T06:45:11.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.034+0000 7fe9d3e00640 1 -- start start 2024-05-27T06:45:11.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d3e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9cc15a620 con 0x7fe9cc154660 2024-05-27T06:45:11.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47028/0 (socket says 172.21.15.81:47028) 2024-05-27T06:45:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d1e00640 1 -- 172.21.15.81:0/4110596221 learned_addr learned my addr 172.21.15.81:0/4110596221 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9cc0c7bf0 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.038+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe9cc0c9ec0 con 0x7fe9cc0c9ab0 2024-05-27T06:45:11.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.042+0000 7fe9d1e00640 1 -- 172.21.15.81:0/4110596221 >> v1:172.21.15.106:6789/0 conn(0x7fe9cc0c96a0 legacy=0x7fe9cc0cff60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T06:45:11.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.042+0000 7fe9d1e00640 1 -- 172.21.15.81:0/4110596221 >> v1:172.21.15.81:6789/0 conn(0x7fe9cc154660 legacy=0x7fe9cc15bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:11.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.042+0000 7fe9d1e00640 1 -- 172.21.15.81:0/4110596221 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc0c7ed0 con 0x7fe9cc0c9ab0 2024-05-27T06:45:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.046+0000 7fe9d1e00640 1 --2- 172.21.15.81:0/4110596221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc069620 tx=0x7fe9b801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58d996b00703009c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.046+0000 7fe9cbe00640 1 -- 172.21.15.81:0/4110596221 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80b1070 con 0x7fe9cc0c9ab0 2024-05-27T06:45:11.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.046+0000 7fe9cbe00640 1 -- 172.21.15.81:0/4110596221 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b80ac410 con 0x7fe9cc0c9ab0 2024-05-27T06:45:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c9ab0 msgr2=0x7fe9cc165fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/4110596221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc069620 tx=0x7fe9b801c540 comp rx=0 tx=0).stop 2024-05-27T06:45:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 shutdown_connections 2024-05-27T06:45:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/4110596221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc165fb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 >> 172.21.15.81:0/4110596221 conn(0x7fe9cc068ab0 msgr2=0x7fe9cc068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 shutdown_connections 2024-05-27T06:45:11.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.050+0000 7fe9d3e00640 1 -- 172.21.15.81:0/4110596221 wait complete. 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.054+0000 7fe9d3e00640 1 Processor -- start 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.054+0000 7fe9d3e00640 1 -- start start 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.054+0000 7fe9d3e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.054+0000 7fe9d3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc1f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47032/0 (socket says 172.21.15.81:47032) 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 -- 172.21.15.81:0/1392237662 learned_addr learned my addr 172.21.15.81:0/1392237662 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9cc154660 0x7fe9cc1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d0a00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc1f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc154a70 con 0x7fe9cc0c9ab0 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc0c9210 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc0d0670 con 0x7fe9cc154660 2024-05-27T06:45:11.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9cc154660 0x7fe9cc1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 -- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9cc154660 msgr2=0x7fe9cc1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9cc154660 0x7fe9cc1502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 -- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9cc0c9ab0 msgr2=0x7fe9cc1f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc1f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc1f5980 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d1400640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009940 tx=0x7fe9c0009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c000f470 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c000a8b0 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c000ac80 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc14f7a0 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.058+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc1550f0 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.062+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9cc154f10 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe9c0022480 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9c9600640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 0x7fe9ac071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:11.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9d0a00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 0x7fe9ac071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:11.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe9c008ed40 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe9cc154f10 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.066+0000 7fe9d0a00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 0x7fe9ac071910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0ce490 tx=0x7fe9c4015040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:11.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.318+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd9f0005860 con 0x7fda1c15c580 2024-05-27T06:45:11.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.322+0000 7fda0fe00640 1 -- 172.21.15.81:0/1941358123 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fda1405c050 con 0x7fda1c15c580 2024-05-27T06:45:11.317 INFO:teuthology.orchestra.run.smithi081.stdout:55834574852 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 msgr2=0x7fd9fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 0x7fd9fc071940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda1001ed70 tx=0x7fda10009040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 msgr2=0x7fda1c158600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 0x7fda1c158600 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda14007d20 tx=0x7fda1400e4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 shutdown_connections 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda1c15c580 0x7fda1c158600 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd9fc06f430 0x7fd9fc071940 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda1001ed70 tx=0x7fda10009040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda1c0c8b30 0x7fda1c1580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 --2- 172.21.15.81:0/1941358123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda1c072700 0x7fda1c14c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 >> 172.21.15.81:0/1941358123 conn(0x7fda1c06fea0 msgr2=0x7fda1c072b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 shutdown_connections 2024-05-27T06:45:11.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.326+0000 7fda0d600640 1 -- 172.21.15.81:0/1941358123 wait complete. 2024-05-27T06:45:11.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.338+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe36004de80 con 0x7fe36004d360 2024-05-27T06:45:11.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.338+0000 7fe35e000640 1 -- 172.21.15.81:0/2346467756 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe35805b570 con 0x7fe36004d360 2024-05-27T06:45:11.333 INFO:teuthology.orchestra.run.smithi081.stdout:55834574852 2024-05-27T06:45:11.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4dd4076dd0 con 0x7f4dd41611d0 2024-05-27T06:45:11.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7f4dd1600640 1 -- 172.21.15.81:0/2860671588 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4dd4161b20 con 0x7f4dd41611d0 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 msgr2=0x7fe3480719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 0x7fe3480719e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe354000c10 tx=0x7fe354008040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 msgr2=0x7fe3600c7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 0x7fe3600c7e40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe35800d5a0 tx=0x7fe35800da60 comp rx=0 tx=0).stop 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 shutdown_connections 2024-05-27T06:45:11.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe34806f4d0 0x7fe3480719e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe36004d360 0x7fe3600c7e40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3600487e0 0x7fe360052740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 --2- 172.21.15.81:0/2346467756 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe360047b70 0x7fe3600cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.342+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 >> 172.21.15.81:0/2346467756 conn(0x7fe36000a000 msgr2=0x7fe360047460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.339 INFO:teuthology.orchestra.run.smithi081.stdout:55834574852 2024-05-27T06:45:11.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 shutdown_connections 2024-05-27T06:45:11.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7fe367200640 1 -- 172.21.15.81:0/2346467756 wait complete. 2024-05-27T06:45:11.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 msgr2=0x7f4db4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 0x7f4db4071940 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc000f90 tx=0x7f4dcc0aa000 comp rx=0 tx=0).stop 2024-05-27T06:45:11.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 msgr2=0x7f4dd40c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 0x7f4dd40c6ad0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4075770 tx=0x7f4dc800c820 comp rx=0 tx=0).stop 2024-05-27T06:45:11.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7faf8004c700 con 0x7faf6c004370 2024-05-27T06:45:11.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 shutdown_connections 2024-05-27T06:45:11.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4dd4073ef0 0x7f4dd4153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4db406f430 0x7f4db4071940 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd41611d0 0x7f4dd40c6ad0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 --2- 172.21.15.81:0/2860671588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4dd40735c0 0x7f4dd4155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.346+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 >> 172.21.15.81:0/2860671588 conn(0x7f4dd406dca0 msgr2=0x7f4dd40c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7faf7ea00640 1 -- 172.21.15.81:0/2317690801 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faf7805b750 con 0x7faf6c004370 2024-05-27T06:45:11.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 shutdown_connections 2024-05-27T06:45:11.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7f4ddc000640 1 -- 172.21.15.81:0/2860671588 wait complete. 2024-05-27T06:45:11.344 INFO:teuthology.orchestra.run.smithi081.stdout:55834574852 2024-05-27T06:45:11.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa774005590 con 0x7fa7980cab10 2024-05-27T06:45:11.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff418005860 con 0x7ff43c073ef0 2024-05-27T06:45:11.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7fa792000640 1 -- 172.21.15.81:0/1551989235 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa78c05be00 con 0x7fa7980cab10 2024-05-27T06:45:11.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 msgr2=0x7faf68071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7ff439600640 1 -- 172.21.15.81:0/252876794 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff42805aea0 con 0x7ff43c073ef0 2024-05-27T06:45:11.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 0x7faf68071960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faf74009ae0 tx=0x7faf74008040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.350+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 msgr2=0x7faf800ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.347 INFO:teuthology.orchestra.run.smithi081.stdout:55834574853 2024-05-27T06:45:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faf7800e780 tx=0x7faf7800ec40 comp rx=0 tx=0).stop 2024-05-27T06:45:11.348 INFO:teuthology.orchestra.run.smithi081.stdout:55834574853 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 shutdown_connections 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7faf6806f450 0x7faf68071960 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf80175950 0x7faf800d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf800d69e0 0x7faf800d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 --2- 172.21.15.81:0/2317690801 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faf6c004370 0x7faf800ce130 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 >> 172.21.15.81:0/2317690801 conn(0x7faf800514d0 msgr2=0x7faf800d3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 shutdown_connections 2024-05-27T06:45:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7faf87e00640 1 -- 172.21.15.81:0/2317690801 wait complete. 2024-05-27T06:45:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 msgr2=0x7fa77c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 0x7fa77c071940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa7880b2000 tx=0x7fa7880aa070 comp rx=0 tx=0).stop 2024-05-27T06:45:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 msgr2=0x7fa79815ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00d620 tx=0x7fa78c00dae0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 msgr2=0x7ff41c0719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 0x7ff41c0719e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff430000f70 tx=0x7ff430001b10 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 msgr2=0x7ff43c0c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c3c0 tx=0x7ff42800c880 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 shutdown_connections 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff43c073ef0 0x7ff43c0c6910 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c3c0 tx=0x7ff42800c880 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff41c06f4d0 0x7ff41c0719e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.354+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff43c1611d0 0x7ff43c0c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7ff444000640 1 --2- 172.21.15.81:0/252876794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff43c0735c0 0x7ff43c1f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 >> 172.21.15.81:0/252876794 conn(0x7ff43c06dca0 msgr2=0x7ff43c0c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 shutdown_connections 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa798164190 0x7fa79814d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa77c06f430 0x7fa77c071940 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7980caf20 0x7fa79814d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 --2- 172.21.15.81:0/1551989235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7980cab10 0x7fa79815ce20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 >> 172.21.15.81:0/1551989235 conn(0x7fa79806f080 msgr2=0x7fa798162a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 shutdown_connections 2024-05-27T06:45:11.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7ff444000640 1 -- 172.21.15.81:0/252876794 wait complete. 2024-05-27T06:45:11.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 shutdown_connections 2024-05-27T06:45:11.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.358+0000 7fa7a0800640 1 -- 172.21.15.81:0/1551989235 wait complete. 2024-05-27T06:45:11.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.2 2024-05-27T06:45:11.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.378+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1f2c005590 con 0x7f1f38003a10 2024-05-27T06:45:11.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.378+0000 7f1f43e00640 1 -- 172.21.15.81:0/1684816462 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f3c016a10 con 0x7f1f38003a10 2024-05-27T06:45:11.373 INFO:teuthology.orchestra.run.smithi081.stdout:55834574853 2024-05-27T06:45:11.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 msgr2=0x7f1f300719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 0x7f1f300719e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1f4804dc90 tx=0x7f1f44008040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 msgr2=0x7f1f48035700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c320 tx=0x7f1f38005840 comp rx=0 tx=0).stop 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 shutdown_connections 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1f48036250 0x7f1f480e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1f3006f4d0 0x7f1f300719e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1f480059a0 0x7f1f480e5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 --2- 172.21.15.81:0/1684816462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f38003a10 0x7f1f48035700 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 >> 172.21.15.81:0/1684816462 conn(0x7f1f480503c0 msgr2=0x7f1f480db2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 shutdown_connections 2024-05-27T06:45:11.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.382+0000 7f1f55800640 1 -- 172.21.15.81:0/1684816462 wait complete. 2024-05-27T06:45:11.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.0 2024-05-27T06:45:11.387 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.1 2024-05-27T06:45:11.388 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.3 2024-05-27T06:45:11.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.398+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fe9cc15d9f0 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.398+0000 7fe9c9600640 1 -- 172.21.15.81:0/1392237662 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe9c000ae20 con 0x7fe9cc0c96a0 2024-05-27T06:45:11.392 INFO:teuthology.orchestra.run.smithi081.stdout:55834574852 2024-05-27T06:45:11.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2024-05-27T06:45:11.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.4 2024-05-27T06:45:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 msgr2=0x7fe9ac071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 0x7fe9ac071910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0ce490 tx=0x7fe9c4015040 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 msgr2=0x7fe9cc1f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009940 tx=0x7fe9c0009e00 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 shutdown_connections 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9cc154660 0x7fe9cc1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe9ac06f400 0x7fe9ac071910 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9cc0c9ab0 0x7fe9cc1f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 --2- 172.21.15.81:0/1392237662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9cc0c96a0 0x7fe9cc1f5440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 >> 172.21.15.81:0/1392237662 conn(0x7fe9cc068ab0 msgr2=0x7fe9cc159960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 shutdown_connections 2024-05-27T06:45:11.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:11.402+0000 7fe9d3e00640 1 -- 172.21.15.81:0/1392237662 wait complete. 2024-05-27T06:45:11.402 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.5 2024-05-27T06:45:11.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.6 2024-05-27T06:45:12.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-27T06:45:12.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-27T06:45:12.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-27T06:45:12.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-27T06:45:12.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-27T06:45:12.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-27T06:45:12.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-27T06:45:12.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-27T06:45:12.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.594+0000 7f1bcf200640 1 Processor -- start 2024-05-27T06:45:12.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.594+0000 7f1bcf200640 1 -- start start 2024-05-27T06:45:12.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcf200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd00742e0 0x7f1bd0072920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcde00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f1bd00722b0 0x7f1bd0159030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59172/0 (socket says 172.21.15.81:59172) 2024-05-27T06:45:12.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcde00640 1 -- 172.21.15.81:0/429157715 learned_addr learned my addr 172.21.15.81:0/429157715 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:12.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd006fe90 con 0x7f1bd0072e60 2024-05-27T06:45:12.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd0070170 con 0x7f1bd00722b0 2024-05-27T06:45:12.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcd400640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd00742e0 0x7f1bd0072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd0158ba0 con 0x7f1bd00742e0 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978557717 0 0) 0x7f1bd006fe90 con 0x7f1bd0072e60 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bb4003530 con 0x7f1bd0072e60 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420571267 0 0) 0x7f1bd0070170 con 0x7f1bd00722b0 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd006fe90 con 0x7f1bd00722b0 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1125014506 0 0) 0x7f1bb4003530 con 0x7f1bd0072e60 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd00742e0 msgr2=0x7f1bd0072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd00742e0 0x7f1bd0072920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 >> v1:172.21.15.106:6789/0 conn(0x7f1bd00722b0 legacy=0x7f1bd0159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd0070450 con 0x7f1bd0072e60 2024-05-27T06:45:12.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f1bc8001a80 con 0x7f1bd0072e60 2024-05-27T06:45:12.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 >> v1:172.21.15.81:6789/0 conn(0x7f1bd0072e60 legacy=0x7f1bd0162fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004c80 0x7f1bb40050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bb40037b0 0x7f1bb4003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bb4004210 0x7f1bb40079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0158ba0 con 0x7f1bb4004c80 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4008100 con 0x7f1bb40037b0 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb4008300 con 0x7f1bb4004210 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcd400640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bb40037b0 0x7f1bb4003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bb4004210 0x7f1bb40079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcde00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004c80 0x7f1bb40050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 -- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bb40037b0 msgr2=0x7f1bb4003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bb40037b0 0x7f1bb4003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 -- 172.21.15.81:0/429157715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004c80 msgr2=0x7f1bb40050d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004c80 0x7f1bb40050d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bce800640 1 -- 172.21.15.81:0/429157715 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb4003530 con 0x7f1bb4004210 2024-05-27T06:45:12.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.598+0000 7f1bcde00640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004c80 0x7f1bb40050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:12.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bce800640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bb4004210 0x7f1bb40079b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006f0a0 tx=0x7f1bc801ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bb09ad6a04ea8b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80b2070 con 0x7f1bb4004210 2024-05-27T06:45:12.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc80a5d50 con 0x7f1bb4004210 2024-05-27T06:45:12.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcca00640 1 -- 172.21.15.81:0/429157715 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc800c0e0 con 0x7f1bb4004210 2024-05-27T06:45:12.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bb4004210 msgr2=0x7f1bb40079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcf200640 1 --2- 172.21.15.81:0/429157715 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bb4004210 0x7f1bb40079b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006f0a0 tx=0x7f1bc801ce60 comp rx=0 tx=0).stop 2024-05-27T06:45:12.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bce800640 1 -- 172.21.15.81:0/429157715 reap_dead start 2024-05-27T06:45:12.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 shutdown_connections 2024-05-27T06:45:12.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.610+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 >> 172.21.15.81:0/429157715 conn(0x7f1bd006cd90 msgr2=0x7f1bd00cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.618+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 shutdown_connections 2024-05-27T06:45:12.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.618+0000 7f1bcf200640 1 -- 172.21.15.81:0/429157715 wait complete. 2024-05-27T06:45:12.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.622+0000 7f1bcf200640 1 Processor -- start 2024-05-27T06:45:12.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.622+0000 7f1bcf200640 1 -- start start 2024-05-27T06:45:12.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.622+0000 7f1bcf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 0x7f1bd01f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.622+0000 7f1bcf200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.622+0000 7f1bcf200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd014df50 0x7f1bd014e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47054/0 (socket says 172.21.15.81:47054) 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 -- 172.21.15.81:0/1630914893 learned_addr learned my addr 172.21.15.81:0/1630914893 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd00c9790 con 0x7f1bb4004210 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd00746f0 con 0x7f1bd0152fb0 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd01de9e0 con 0x7f1bd014df50 2024-05-27T06:45:12.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bce800640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd014df50 0x7f1bd014e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcde00640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 0x7f1bd01f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 -- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd014df50 msgr2=0x7f1bd014e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd014df50 0x7f1bd014e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 -- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 msgr2=0x7f1bd01f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 0x7f1bd01f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd01f2720 con 0x7f1bd0152fb0 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcde00640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 0x7f1bd01f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:12.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bcd400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0011890 tx=0x7f1bc0011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0019070 con 0x7f1bd0152fb0 2024-05-27T06:45:12.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.626+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc0004240 con 0x7f1bd0152fb0 2024-05-27T06:45:12.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.630+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd01e1100 con 0x7f1bd0152fb0 2024-05-27T06:45:12.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.630+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd01e1680 con 0x7f1bd0152fb0 2024-05-27T06:45:12.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.630+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc000f820 con 0x7f1bd0152fb0 2024-05-27T06:45:12.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.630+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd0050870 con 0x7f1bd0152fb0 2024-05-27T06:45:12.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.634+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1bc001c070 con 0x7f1bd0152fb0 2024-05-27T06:45:12.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.634+0000 7f1bc6000640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 0x7f1bb0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.634+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1bc008fa80 con 0x7f1bd0152fb0 2024-05-27T06:45:12.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.638+0000 7f1bcde00640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 0x7f1bb0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.638+0000 7f1bcde00640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 0x7f1bb0071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc001ec0 tx=0x7f1bbc001d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.638+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1bc005c300 con 0x7f1bd0152fb0 2024-05-27T06:45:12.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.674+0000 7f0e2f200640 1 Processor -- start 2024-05-27T06:45:12.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.674+0000 7f0e2f200640 1 -- start start 2024-05-27T06:45:12.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.674+0000 7f52a2400640 1 Processor -- start 2024-05-27T06:45:12.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.674+0000 7f52a2400640 1 -- start start 2024-05-27T06:45:12.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f3fa9800640 1 Processor -- start 2024-05-27T06:45:12.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0e301545f0 0x7f0e300c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2f200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0e30071f90 0x7f0e300723a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2f200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e300c7780 con 0x7f0e300744a0 2024-05-27T06:45:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0e300c6680 con 0x7f0e301545f0 2024-05-27T06:45:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2f200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0e300c6ff0 con 0x7f0e30071f90 2024-05-27T06:45:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f3fa9800640 1 -- start start 2024-05-27T06:45:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f52a2400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f52a2400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f529c0c6a50 0x7f529c0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f52a2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f529c0cdc20 con 0x7f529c0ce010 2024-05-27T06:45:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f52a2400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f529c15d620 con 0x7f529c0c6a50 2024-05-27T06:45:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f52a2400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f529c160bc0 con 0x7f529c15d0c0 2024-05-27T06:45:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2e800640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f0e300744a0 0x7f0e300748b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59188/0 (socket says 172.21.15.81:59188) 2024-05-27T06:45:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2e800640 1 -- 172.21.15.81:0/691257088 learned_addr learned my addr 172.21.15.81:0/691257088 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2d400640 1 --2- 172.21.15.81:0/691257088 >> v2:172.21.15.106:3300/0 conn(0x7f0e30071f90 0x7f0e300723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2de00640 1 --2- 172.21.15.81:0/691257088 >> v2:172.21.15.81:3300/0 conn(0x7f0e301545f0 0x7f0e300c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55052/0 (socket says 172.21.15.81:55052) 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 -- 172.21.15.81:0/3264430658 learned_addr learned my addr 172.21.15.81:0/3264430658 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529be00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f529c0c6a50 0x7f529c0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2ca00640 1 -- 172.21.15.81:0/691257088 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417045512 0 0) 0x7f0e20001120 con 0x7f0e300744a0 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2ca00640 1 -- 172.21.15.81:0/691257088 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e14003530 con 0x7f0e300744a0 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2de00640 1 -- 172.21.15.81:0/691257088 >> v2:172.21.15.106:3300/0 conn(0x7f0e30071f90 msgr2=0x7f0e300723a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2de00640 1 --2- 172.21.15.81:0/691257088 >> v2:172.21.15.106:3300/0 conn(0x7f0e30071f90 0x7f0e300723a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 -- 172.21.15.81:0/3264430658 >> v2:172.21.15.106:3300/0 conn(0x7f529c0c6a50 msgr2=0x7f529c0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 --2- 172.21.15.81:0/3264430658 >> v2:172.21.15.106:3300/0 conn(0x7f529c0c6a50 0x7f529c0d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 -- 172.21.15.81:0/3264430658 >> v1:172.21.15.81:6789/0 conn(0x7f529c0ce010 legacy=0x7f529c156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f529aa00640 1 -- 172.21.15.81:0/3264430658 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c15fbe0 con 0x7f529c15d0c0 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2de00640 1 -- 172.21.15.81:0/691257088 >> v1:172.21.15.106:6789/0 conn(0x7f0e300744a0 legacy=0x7f0e300748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.678+0000 7f0e2de00640 1 -- 172.21.15.81:0/691257088 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e300c72d0 con 0x7f0e301545f0 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa9800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3fa40d2060 0x7f3fa40d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa9800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fa40cd4e0 con 0x7f3fa40c6a70 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa9800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3fa415d540 con 0x7f3fa40d2060 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa9800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa40cd780 con 0x7f3fa40c8130 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55060/0 (socket says 172.21.15.81:55060) 2024-05-27T06:45:12.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 -- 172.21.15.81:0/2050342503 learned_addr learned my addr 172.21.15.81:0/2050342503 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 -- 172.21.15.81:0/2050342503 >> v2:172.21.15.81:3300/0 conn(0x7f3fa40d2060 msgr2=0x7f3fa40d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f529aa00640 1 --2- 172.21.15.81:0/3264430658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f528c009870 tx=0x7f528c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4bca3e3866834f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2de00640 1 --2- 172.21.15.81:0/691257088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e300c95c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e300ccec0 tx=0x7f0e2801f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4ca510f2a3f16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 --2- 172.21.15.81:0/2050342503 >> v2:172.21.15.81:3300/0 conn(0x7f3fa40d2060 0x7f3fa40d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 -- 172.21.15.81:0/2050342503 >> v1:172.21.15.81:6789/0 conn(0x7f3fa40c6a70 legacy=0x7f3fa4156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.682+0000 7f3fa8e00640 1 -- 172.21.15.81:0/2050342503 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa40cda60 con 0x7f3fa40c8130 2024-05-27T06:45:12.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f529a000640 1 -- 172.21.15.81:0/3264430658 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c0b0070 con 0x7f529c15d0c0 2024-05-27T06:45:12.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f529a000640 1 -- 172.21.15.81:0/3264430658 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528c0a4e50 con 0x7f529c15d0c0 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f529a000640 1 -- 172.21.15.81:0/3264430658 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c0ab820 con 0x7f529c15d0c0 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 -- 172.21.15.81:0/3264430658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 msgr2=0x7f529c160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 --2- 172.21.15.81:0/3264430658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f528c009870 tx=0x7f528c01c440 comp rx=0 tx=0).stop 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2ca00640 1 -- 172.21.15.81:0/691257088 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e280b4070 con 0x7f0e301545f0 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2ca00640 1 -- 172.21.15.81:0/691257088 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e280af600 con 0x7f0e301545f0 2024-05-27T06:45:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2ca00640 1 -- 172.21.15.81:0/691257088 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e280af9d0 con 0x7f0e301545f0 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 -- 172.21.15.81:0/3264430658 shutdown_connections 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 --2- 172.21.15.81:0/3264430658 >> v2:172.21.15.106:3300/0 conn(0x7f529c0c6a50 0x7f529c0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 --2- 172.21.15.81:0/3264430658 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c15d0c0 0x7f529c160680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 -- 172.21.15.81:0/3264430658 >> 172.21.15.81:0/3264430658 conn(0x7f529c06e7b0 msgr2=0x7f529c06ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2f200640 1 -- 172.21.15.81:0/691257088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 msgr2=0x7f0e300c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f0e2f200640 1 --2- 172.21.15.81:0/691257088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e300c95c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e300ccec0 tx=0x7f0e2801f110 comp rx=0 tx=0).stop 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 -- 172.21.15.81:0/3264430658 shutdown_connections 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 -- 172.21.15.81:0/691257088 shutdown_connections 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 --2- 172.21.15.81:0/691257088 >> v2:172.21.15.106:3300/0 conn(0x7f0e30071f90 0x7f0e300723a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 --2- 172.21.15.81:0/691257088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e300c95c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 -- 172.21.15.81:0/691257088 >> 172.21.15.81:0/691257088 conn(0x7f0e30068ab0 msgr2=0x7f0e30068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 -- 172.21.15.81:0/691257088 shutdown_connections 2024-05-27T06:45:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.686+0000 7f52a2400640 1 -- 172.21.15.81:0/3264430658 wait complete. 2024-05-27T06:45:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa8e00640 1 --2- 172.21.15.81:0/2050342503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009870 tx=0x7f3f9401ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18313154aa771c0a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7ff219800640 1 Processor -- start 2024-05-27T06:45:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa2a00640 1 -- 172.21.15.81:0/2050342503 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b5070 con 0x7f3fa40c8130 2024-05-27T06:45:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f0e2f200640 1 -- 172.21.15.81:0/691257088 wait complete. 2024-05-27T06:45:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa2a00640 1 -- 172.21.15.81:0/2050342503 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f940a89f0 con 0x7f3fa40c8130 2024-05-27T06:45:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 -- 172.21.15.81:0/2050342503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 msgr2=0x7f3fa40c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa2a00640 1 -- 172.21.15.81:0/2050342503 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b5070 con 0x7f3fa40c8130 2024-05-27T06:45:12.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 --2- 172.21.15.81:0/2050342503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009870 tx=0x7f3f9401ed70 comp rx=0 tx=0).stop 2024-05-27T06:45:12.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 -- 172.21.15.81:0/2050342503 shutdown_connections 2024-05-27T06:45:12.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7ff219800640 1 -- start start 2024-05-27T06:45:12.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 --2- 172.21.15.81:0/2050342503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa40c8540 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 --2- 172.21.15.81:0/2050342503 >> v2:172.21.15.81:3300/0 conn(0x7f3fa40d2060 0x7f3fa40d2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7f3fa9800640 1 -- 172.21.15.81:0/2050342503 >> 172.21.15.81:0/2050342503 conn(0x7f3fa406e980 msgr2=0x7f3fa406ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.690+0000 7fc539400640 1 Processor -- start 2024-05-27T06:45:12.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f3fa9800640 1 -- 172.21.15.81:0/2050342503 shutdown_connections 2024-05-27T06:45:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7fc539400640 1 -- start start 2024-05-27T06:45:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f3fa9800640 1 -- 172.21.15.81:0/2050342503 wait complete. 2024-05-27T06:45:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 Processor -- start 2024-05-27T06:45:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 -- start start 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f0e2f200640 1 Processor -- start 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff219800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff21406f910 0x7ff21406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff219800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 0x7ff21415c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff213e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff2140c9310 0x7ff2140c6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59194/0 (socket says 172.21.15.81:59194) 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff213e00640 1 -- 172.21.15.81:0/2467601558 learned_addr learned my addr 172.21.15.81:0/2467601558 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:12.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff218e00640 1 --2- 172.21.15.81:0/2467601558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 0x7ff21415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ce010 0x7f529c1fddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 0x7f529c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f529c1697e0 con 0x7f529c15d0c0 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f529c160160 con 0x7f529c0c6a50 2024-05-27T06:45:12.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f52a2400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f529c169960 con 0x7f529c0ce010 2024-05-27T06:45:12.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2140c4ba0 con 0x7ff2140c9310 2024-05-27T06:45:12.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff2141587e0 con 0x7ff21406f910 2024-05-27T06:45:12.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff214158ac0 con 0x7ff21415bf60 2024-05-27T06:45:12.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661263628 0 0) 0x7ff2140c4ba0 con 0x7ff2140c9310 2024-05-27T06:45:12.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1f8003530 con 0x7ff2140c9310 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f529b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.694+0000 7f529aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ce010 0x7f529c1fddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47078/0 (socket says 172.21.15.81:47078) 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 -- 172.21.15.81:0/1129168762 learned_addr learned my addr 172.21.15.81:0/1129168762 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529be00640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 0x7f529c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f3fa9800640 1 Processor -- start 2024-05-27T06:45:12.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7ff218e00640 1 -- 172.21.15.81:0/2467601558 >> v2:172.21.15.106:3300/0 conn(0x7ff21406f910 msgr2=0x7ff21406fd20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7ff218e00640 1 --2- 172.21.15.81:0/2467601558 >> v2:172.21.15.106:3300/0 conn(0x7ff21406f910 0x7ff21406fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7ff218e00640 1 -- 172.21.15.81:0/2467601558 >> v1:172.21.15.106:6789/0 conn(0x7ff2140c9310 legacy=0x7ff2140c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 <== mon.1 v1:172.21.15.106:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2019399273 0 0) 0x7ff1f8003530 con 0x7ff2140c9310 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7ff218e00640 1 -- 172.21.15.81:0/2467601558 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff214158da0 con 0x7ff21415bf60 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f3fa9800640 1 -- start start 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 -- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ce010 msgr2=0x7f529c1fddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ce010 0x7f529c1fddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc539400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc534154750 0x7fc5340c7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc539400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc53415e040 0x7fc534164fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc539400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5340c8a20 con 0x7fc53415dc30 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc539400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fc534154b60 con 0x7fc53415e040 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc539400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc5340c8740 con 0x7fc534154750 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 -- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 msgr2=0x7f529c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 0x7f529c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 -- 172.21.15.81:0/1129168762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529c1fe2f0 con 0x7f529c0c6a50 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529be00640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 0x7f529c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc53415e040 0x7fc534164fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc53415e040 0x7fc534164fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57506/0 (socket says 172.21.15.81:57506) 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 -- 172.21.15.81:0/634505449 learned_addr learned my addr 172.21.15.81:0/634505449 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc532800640 1 --2- 172.21.15.81:0/634505449 >> v2:172.21.15.106:3300/0 conn(0x7fc534154750 0x7fc5340c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 -- 172.21.15.81:0/634505449 >> v2:172.21.15.106:3300/0 conn(0x7fc534154750 msgr2=0x7fc5340c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 --2- 172.21.15.81:0/634505449 >> v2:172.21.15.106:3300/0 conn(0x7fc534154750 0x7fc5340c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 -- 172.21.15.81:0/634505449 >> v1:172.21.15.81:6789/0 conn(0x7fc53415dc30 legacy=0x7fc534074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f83df200640 1 Processor -- start 2024-05-27T06:45:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f529b400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f528800b3e0 tx=0x7f528800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f83df200640 1 -- start start 2024-05-27T06:45:12.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7fc533200640 1 -- 172.21.15.81:0/634505449 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53415a7c0 con 0x7fc53415e040 2024-05-27T06:45:12.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288015020 con 0x7f529c0c6a50 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5288011850 con 0x7f529c0c6a50 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288011c40 con 0x7f529c0c6a50 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529c14ecf0 con 0x7f529c0c6a50 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.698+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f529c149d70 con 0x7f529c0c6a50 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 0x7f3fa41f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83df200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83e0152b90 0x7f83e00c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f529c050870 con 0x7f529c0c6a50 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55082/0 (socket says 172.21.15.81:55082) 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa8e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa8e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47084/0 (socket says 172.21.15.81:47084) 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f3fa8e00640 1 -- 172.21.15.81:0/692928375 learned_addr learned my addr 172.21.15.81:0/692928375 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83dde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83e0152b90 0x7f83e00c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83dde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83e0152b90 0x7f83e00c7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57516/0 (socket says 172.21.15.81:57516) 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83dde00640 1 -- 172.21.15.81:0/2509178791 learned_addr learned my addr 172.21.15.81:0/2509178791 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2f200640 1 -- start start 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 0x7f0e3014fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e30143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e00c9640 con 0x7f83e015c930 2024-05-27T06:45:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 0x7f0e3014fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 0x7f0e3014fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47088/0 (socket says 172.21.15.81:47088) 2024-05-27T06:45:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e00c9920 con 0x7f83e00c7750 2024-05-27T06:45:12.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f83e00c93f0 con 0x7f83e0152b90 2024-05-27T06:45:12.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7fc533200640 1 --2- 172.21.15.81:0/634505449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc53415e040 0x7fc534164fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc53406f320 tx=0x7fc52c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d9bf7671eba7ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7f0e2d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55092/0 (socket says 172.21.15.81:55092) 2024-05-27T06:45:12.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.702+0000 7ff218e00640 1 --2- 172.21.15.81:0/2467601558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 0x7ff21415c370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00ac30 tx=0x7ff20c01f4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0856370487518df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f83dde00640 1 -- 172.21.15.81:0/2509178791 >> v1:172.21.15.106:6789/0 conn(0x7f83e00c7750 legacy=0x7f83e015c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0b4070 con 0x7ff21415bf60 2024-05-27T06:45:12.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc531e00640 1 -- 172.21.15.81:0/634505449 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0a57a0 con 0x7fc53415e040 2024-05-27T06:45:12.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc531e00640 1 -- 172.21.15.81:0/634505449 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52c0a5940 con 0x7fc53415e040 2024-05-27T06:45:12.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc531e00640 1 -- 172.21.15.81:0/634505449 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0a5d10 con 0x7fc53415e040 2024-05-27T06:45:12.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f83dde00640 1 -- 172.21.15.81:0/2509178791 >> v1:172.21.15.81:6789/0 conn(0x7f83e015c930 legacy=0x7f83e006f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2de00640 1 -- 172.21.15.81:0/256336383 learned_addr learned my addr 172.21.15.81:0/256336383 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300cd1f0 con 0x7f0e301545f0 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0e30154a00 con 0x7f0e30071f90 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0e300cbff0 con 0x7f0e300744a0 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20c0af400 con 0x7ff21415bf60 2024-05-27T06:45:12.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff213400640 1 -- 172.21.15.81:0/2467601558 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204000ee0 con 0x7ff21415bf60 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5288018020 con 0x7f529c0c6a50 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40c8a80 con 0x7f3fa40c6a70 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40cd270 con 0x7f3fa40d2060 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa415d120 con 0x7f3fa40c8130 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa3e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 0x7f3fa41f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f83dde00640 1 -- 172.21.15.81:0/2509178791 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e00c9c00 con 0x7f83e0152b90 2024-05-27T06:45:12.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 -- 172.21.15.81:0/634505449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc53415e040 msgr2=0x7fc534164fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 --2- 172.21.15.81:0/634505449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc53415e040 0x7fc534164fc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc53406f320 tx=0x7fc52c01c9f0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 -- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 msgr2=0x7f0e3014fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 0x7f0e3014fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 -- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 msgr2=0x7f0e30143430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e30143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e300c8460 con 0x7f0e300744a0 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 -- 172.21.15.81:0/634505449 shutdown_connections 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 --2- 172.21.15.81:0/634505449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc53415e040 0x7fc534164fc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 --2- 172.21.15.81:0/634505449 >> v2:172.21.15.106:3300/0 conn(0x7fc534154750 0x7fc5340c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 -- 172.21.15.81:0/634505449 >> 172.21.15.81:0/634505449 conn(0x7fc53406cd90 msgr2=0x7fc5340cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2d400640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00e8a0 tx=0x7f0e1c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 -- 172.21.15.81:0/634505449 shutdown_connections 2024-05-27T06:45:12.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7fc539400640 1 -- 172.21.15.81:0/634505449 wait complete. 2024-05-27T06:45:12.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 -- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 msgr2=0x7f3fa4150120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 msgr2=0x7ff21415c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 --2- 172.21.15.81:0/2467601558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 0x7ff21415c370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00ac30 tx=0x7ff20c01f4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c017070 con 0x7f0e300744a0 2024-05-27T06:45:12.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 shutdown_connections 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 --2- 172.21.15.81:0/2467601558 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21415bf60 0x7ff21415c370 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 --2- 172.21.15.81:0/2467601558 >> v2:172.21.15.106:3300/0 conn(0x7ff21406f910 0x7ff21406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 >> 172.21.15.81:0/2467601558 conn(0x7ff21406d150 msgr2=0x7ff2140cd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 -- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 msgr2=0x7f3fa41f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f5293400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 0x7f527c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 0x7f3fa41f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e1c00c940 con 0x7f0e300744a0 2024-05-27T06:45:12.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c009e60 con 0x7f0e300744a0 2024-05-27T06:45:12.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e300ce800 con 0x7f0e300744a0 2024-05-27T06:45:12.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa41f3940 con 0x7f3fa40d2060 2024-05-27T06:45:12.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa3e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 0x7f3fa41f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:12.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa3400640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:12.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3006ecc0 con 0x7f0e300744a0 2024-05-27T06:45:12.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f528808f4d0 con 0x7f529c0c6a50 2024-05-27T06:45:12.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.706+0000 7f3fa8e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f9401f610 tx=0x7f3f9401bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 shutdown_connections 2024-05-27T06:45:12.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f529aa00640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 0x7f527c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7ff219800640 1 -- 172.21.15.81:0/2467601558 wait complete. 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f940b5040 con 0x7f3fa40d2060 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f940c0070 con 0x7f3fa40d2060 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94003a60 con 0x7f3fa40d2060 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa41551c0 con 0x7f3fa40d2060 2024-05-27T06:45:12.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 Processor -- start 2024-05-27T06:45:12.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41571e0 con 0x7f3fa40d2060 2024-05-27T06:45:12.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 -- start start 2024-05-27T06:45:12.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f528801f020 con 0x7f529c0c6a50 2024-05-27T06:45:12.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f83dde00640 1 --2- 172.21.15.81:0/2509178791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e0152b90 0x7f83e00c7210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0d77cf20364c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f529aa00640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 0x7f527c071940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f528c009870 tx=0x7f528c0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 0x7fc534153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc53415e040 0x7fc53414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0e1c01a070 con 0x7f0e300744a0 2024-05-27T06:45:12.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e30050870 con 0x7f0e300744a0 2024-05-27T06:45:12.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc53406d5a0 con 0x7fc534154750 2024-05-27T06:45:12.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f0e26000640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 0x7f0e10071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83dca00640 1 -- 172.21.15.81:0/2509178791 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00b1070 con 0x7f83e0152b90 2024-05-27T06:45:12.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.710+0000 7fc539400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc53415b580 con 0x7fc53415dc30 2024-05-27T06:45:12.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83dca00640 1 -- 172.21.15.81:0/2509178791 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d00ac600 con 0x7f83e0152b90 2024-05-27T06:45:12.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc539400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc53406d720 con 0x7fc53415e040 2024-05-27T06:45:12.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f0e2de00640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 0x7f0e10071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e1c057f00 con 0x7f0e300744a0 2024-05-27T06:45:12.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4157830 con 0x7f3fa40d2060 2024-05-27T06:45:12.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3f94003c00 con 0x7f3fa40d2060 2024-05-27T06:45:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83dca00640 1 -- 172.21.15.81:0/2509178791 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00ac9d0 con 0x7f83e0152b90 2024-05-27T06:45:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47090/0 (socket says 172.21.15.81:47090) 2024-05-27T06:45:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 -- 172.21.15.81:0/3682802569 learned_addr learned my addr 172.21.15.81:0/3682802569 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc533200640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc53415e040 0x7fc53414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc532800640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 0x7fc534153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 -- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc53415e040 msgr2=0x7fc53414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc53415e040 0x7fc53414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 -- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 msgr2=0x7fc534153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 0x7fc534153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 -- 172.21.15.81:0/3682802569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc534164fe0 con 0x7fc53415dc30 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc532800640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 0x7fc534153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7fc52be00640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c420 tx=0x7fc51c00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f0e2de00640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 0x7f0e10071960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0e2801f150 tx=0x7f0e280aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f3f9be00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 0x7f3f84071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3f9412c7f0 con 0x7f3fa40d2060 2024-05-27T06:45:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f3fa3e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 0x7f3f84071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e0152b90 msgr2=0x7f83e00c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 --2- 172.21.15.81:0/2509178791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e0152b90 0x7f83e00c7210 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d001c4d0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 shutdown_connections 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 --2- 172.21.15.81:0/2509178791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e0152b90 0x7f83e00c7210 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 >> 172.21.15.81:0/2509178791 conn(0x7f83e006cd90 msgr2=0x7f83e00cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 shutdown_connections 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7f83df200640 1 -- 172.21.15.81:0/2509178791 wait complete. 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.714+0000 7ff219800640 1 Processor -- start 2024-05-27T06:45:12.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7f3fa3e00640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 0x7f3f84071a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f900098b0 tx=0x7f3f90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 -- start start 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21406f910 0x7ff21415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff21415bf60 0x7ff214151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff21415c8b0 con 0x7ff2140c9310 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff214159360 con 0x7ff21415bf60 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff219800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff2140c9720 con 0x7ff21406f910 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff218e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff21415bf60 0x7ff214151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57540/0 (socket says 172.21.15.81:57540) 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 -- 172.21.15.81:0/31653933 learned_addr learned my addr 172.21.15.81:0/31653933 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 -- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21406f910 msgr2=0x7ff21415a6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:45:12.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21406f910 0x7ff21415a6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 -- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff21415bf60 msgr2=0x7ff214151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff21415bf60 0x7ff214151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.718+0000 7ff20be00640 1 -- 172.21.15.81:0/31653933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21414f0d0 con 0x7ff2140c9310 2024-05-27T06:45:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff20be00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00b410 tx=0x7ff1fc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 Processor -- start 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc015020 con 0x7ff2140c9310 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1fc011830 con 0x7ff2140c9310 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff219800640 1 -- 172.21.15.81:0/31653933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21414f2e0 con 0x7ff2140c9310 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff219800640 1 -- 172.21.15.81:0/31653933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21414aa50 con 0x7ff2140c9310 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc011c20 con 0x7ff2140c9310 2024-05-27T06:45:12.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 -- start start 2024-05-27T06:45:12.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec0052a0 con 0x7ff2140c9310 2024-05-27T06:45:12.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83e0152b90 0x7f83e013a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 0x7f83e013a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0166bb0 con 0x7f83e015c930 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0166a30 con 0x7f83e00c7750 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83df200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f83e0166d30 con 0x7f83e0152b90 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dde00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83e0152b90 0x7f83e013a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dde00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:47104/0 (socket says 172.21.15.81:47104) 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dde00640 1 -- 172.21.15.81:0/3187765771 learned_addr learned my addr 172.21.15.81:0/3187765771 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83de800640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 0x7f83e013a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dde00640 1 -- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83e0152b90 msgr2=0x7f83e013a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.722+0000 7f83dde00640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83e0152b90 0x7f83e013a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83dde00640 1 -- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 msgr2=0x7f83e013a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83dde00640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 0x7f83e013a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83dde00640 1 -- 172.21.15.81:0/3187765771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e0154010 con 0x7f83e00c7750 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83de800640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 0x7f83e013a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0e1c05b910 con 0x7f0e300744a0 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c017070 con 0x7fc53415dc30 2024-05-27T06:45:12.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc51c0052d0 con 0x7fc53415dc30 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0054c0 con 0x7fc53415dc30 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83dde00640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83d00190e0 tx=0x7f83d001ad10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00bb020 con 0x7f83e00c7750 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d00b7620 con 0x7f83e00c7750 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00b1040 con 0x7f83e00c7750 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e0141d50 con 0x7f83e00c7750 2024-05-27T06:45:12.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e01530c0 con 0x7f83e00c7750 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff1fc018020 con 0x7ff2140c9310 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7ff210c00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 0x7ff1f00719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff1fc08f6b0 con 0x7ff2140c9310 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc534163170 con 0x7fc53415dc30 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.726+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5341f92d0 con 0x7fc53415dc30 2024-05-27T06:45:12.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7f4441800640 1 Processor -- start 2024-05-27T06:45:12.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5341f90e0 con 0x7fc53415dc30 2024-05-27T06:45:12.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7f4441800640 1 -- start start 2024-05-27T06:45:12.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3f940188d0 con 0x7f3fa40d2060 2024-05-27T06:45:12.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e0050870 con 0x7f83e00c7750 2024-05-27T06:45:12.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7ff213e00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 0x7ff1f00719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7ff213e00640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 0x7ff1f00719e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff204001050 tx=0x7ff204004c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc51c01a070 con 0x7fc53415dc30 2024-05-27T06:45:12.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f83d00be020 con 0x7f83e00c7750 2024-05-27T06:45:12.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.730+0000 7fc52aa00640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 0x7fc510071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc51c08e8d0 con 0x7fc53415dc30 2024-05-27T06:45:12.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f83d6000640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 0x7f83c00719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f83d0128e80 con 0x7f83e00c7750 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f4441800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f443c152c80 0x7f443c159390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f4441800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f4441800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f443c0c4a50 con 0x7f443c06fda0 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f4441800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f443c0c5880 con 0x7f443c152c80 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f4441800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f443c0c41d0 con 0x7f443c0c8e30 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7fc532800640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 0x7fc510071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f83dd400640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 0x7f83c00719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc51c05b150 con 0x7fc53415dc30 2024-05-27T06:45:12.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f443aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f443c152c80 0x7f443c159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f443aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f443c152c80 0x7f443c159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57552/0 (socket says 172.21.15.81:57552) 2024-05-27T06:45:12.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f443aa00640 1 -- 172.21.15.81:0/3083545851 learned_addr learned my addr 172.21.15.81:0/3083545851 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7fc532800640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 0x7fc510071990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5340c4ad0 tx=0x7fc524008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7f83dd400640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 0x7f83c00719e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83e014cb00 tx=0x7f83cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.734+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff1fc01f020 con 0x7ff2140c9310 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443a000640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f4439600640 1 -- 172.21.15.81:0/3083545851 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723599571 0 0) 0x7f443c0c4a50 con 0x7f443c06fda0 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443aa00640 1 -- 172.21.15.81:0/3083545851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 msgr2=0x7f443c0c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443aa00640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c0c9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443aa00640 1 -- 172.21.15.81:0/3083545851 >> v1:172.21.15.81:6789/0 conn(0x7f443c06fda0 legacy=0x7f443c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443aa00640 1 -- 172.21.15.81:0/3083545851 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c0c4490 con 0x7f443c152c80 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f443a000640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c0c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:12.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.738+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f83d00c5060 con 0x7f83e00c7750 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.742+0000 7f443aa00640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c159390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a66a010e593b71e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4439600640 1 -- 172.21.15.81:0/3083545851 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340a56e0 con 0x7f443c152c80 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4439600640 1 -- 172.21.15.81:0/3083545851 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44340a5880 con 0x7f443c152c80 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 -- 172.21.15.81:0/3083545851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 msgr2=0x7f443c159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4439600640 1 -- 172.21.15.81:0/3083545851 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44340a56e0 con 0x7f443c152c80 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c159390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401ca30 comp rx=0 tx=0).stop 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 -- 172.21.15.81:0/3083545851 shutdown_connections 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c0c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 --2- 172.21.15.81:0/3083545851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c159390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.746+0000 7f4441800640 1 -- 172.21.15.81:0/3083545851 >> 172.21.15.81:0/3083545851 conn(0x7f443c06cd90 msgr2=0x7f443c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.750+0000 7f4441800640 1 -- 172.21.15.81:0/3083545851 shutdown_connections 2024-05-27T06:45:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.750+0000 7f4441800640 1 -- 172.21.15.81:0/3083545851 wait complete. 2024-05-27T06:45:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 Processor -- start 2024-05-27T06:45:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 -- start start 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 0x7f443c073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c14e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f443c153090 con 0x7f443c152c80 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f443c0c8910 con 0x7f443c06fda0 2024-05-27T06:45:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f4441800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f443c0705c0 con 0x7f443c0c8e30 2024-05-27T06:45:12.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f443b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f443b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57568/0 (socket says 172.21.15.81:57568) 2024-05-27T06:45:12.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.754+0000 7f443b400640 1 -- 172.21.15.81:0/2631937994 learned_addr learned my addr 172.21.15.81:0/2631937994 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:12.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443aa00640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 0x7f443c073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 -- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 msgr2=0x7f443c14e4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c14e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 -- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 msgr2=0x7f443c073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 0x7f443c073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 -- 172.21.15.81:0/2631937994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f443c1f5f80 con 0x7f443c152c80 2024-05-27T06:45:12.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443aa00640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 0x7f443c073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f443b400640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3f0 tx=0x7f442800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428017070 con 0x7f443c152c80 2024-05-27T06:45:12.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4428003600 con 0x7f443c152c80 2024-05-27T06:45:12.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.758+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44280039f0 con 0x7f443c152c80 2024-05-27T06:45:12.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.762+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f443c167c80 con 0x7f443c152c80 2024-05-27T06:45:12.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.762+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f443c1f64e0 con 0x7f443c152c80 2024-05-27T06:45:12.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.766+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f442801a070 con 0x7f443c152c80 2024-05-27T06:45:12.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.766+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f443c06a560 con 0x7f443c152c80 2024-05-27T06:45:12.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.766+0000 7f4432a00640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 0x7f441c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:12.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.766+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4428057380 con 0x7f443c152c80 2024-05-27T06:45:12.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.770+0000 7f443aa00640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 0x7f441c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.774+0000 7f443aa00640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 0x7f441c071910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f44340ae000 tx=0x7f44340a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.778+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f443c1f5f80 con 0x7f443c152c80 2024-05-27T06:45:12.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.974+0000 7f1bcf200640 1 -- 172.21.15.81:0/1630914893 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1bd00737a0 con 0x7f1bd0152fb0 2024-05-27T06:45:12.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.974+0000 7f1bc6000640 1 -- 172.21.15.81:0/1630914893 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1bc0060600 con 0x7f1bd0152fb0 2024-05-27T06:45:12.969 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:12.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 msgr2=0x7f1bb0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 0x7f1bb0071910 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc001ec0 tx=0x7f1bbc001d70 comp rx=0 tx=0).stop 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 msgr2=0x7f1bd01e1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0011890 tx=0x7f1bc0011c90 comp rx=0 tx=0).stop 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 shutdown_connections 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1bd014df50 0x7f1bd014e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bb006f400 0x7f1bb0071910 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd0152fb0 0x7f1bd01e1f40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 --2- 172.21.15.81:0/1630914893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1bb4004210 0x7f1bd01f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:12.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 >> 172.21.15.81:0/1630914893 conn(0x7f1bd006cd90 msgr2=0x7f1bd00c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:12.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 shutdown_connections 2024-05-27T06:45:12.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:12.978+0000 7f1bbb400640 1 -- 172.21.15.81:0/1630914893 wait complete. 2024-05-27T06:45:13.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-05-27T06:45:13.008 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.038+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3fa4154f50 con 0x7f3fa40d2060 2024-05-27T06:45:13.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3f9be00640 1 -- 172.21.15.81:0/692928375 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3f940f97e0 con 0x7f3fa40d2060 2024-05-27T06:45:13.035 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:13.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 msgr2=0x7f3f84071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 0x7f3f84071a10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f900098b0 tx=0x7f3f90008040 comp rx=0 tx=0).stop 2024-05-27T06:45:13.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 msgr2=0x7f3fa4149400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f9401f610 tx=0x7f3f9401bb10 comp rx=0 tx=0).stop 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 shutdown_connections 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3fa40c8130 0x7f3fa4150120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f8406f500 0x7f3f84071a10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3fa40d2060 0x7f3fa4149400 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 --2- 172.21.15.81:0/692928375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa40c6a70 0x7f3fa41f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 >> 172.21.15.81:0/692928375 conn(0x7f3fa406e980 msgr2=0x7f3fa40cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 shutdown_connections 2024-05-27T06:45:13.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.042+0000 7f3fa9800640 1 -- 172.21.15.81:0/692928375 wait complete. 2024-05-27T06:45:13.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.058+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f529c15cea0 con 0x7f529c0c6a50 2024-05-27T06:45:13.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.058+0000 7f5293400640 1 -- 172.21.15.81:0/1129168762 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f528805c430 con 0x7f529c0c6a50 2024-05-27T06:45:13.053 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:13.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 msgr2=0x7f527c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 0x7f527c071940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f528c009870 tx=0x7f528c0a6000 comp rx=0 tx=0).stop 2024-05-27T06:45:13.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 msgr2=0x7f529c152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f528800b3e0 tx=0x7f528800b8a0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 shutdown_connections 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f529c0ce010 0x7f529c1fddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f527c06f430 0x7f527c071940 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f529c15d0c0 0x7f529c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 --2- 172.21.15.81:0/1129168762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f529c0c6a50 0x7f529c152240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 >> 172.21.15.81:0/1129168762 conn(0x7f529c06e7b0 msgr2=0x7f529c0b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 shutdown_connections 2024-05-27T06:45:13.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.062+0000 7f52a2400640 1 -- 172.21.15.81:0/1129168762 wait complete. 2024-05-27T06:45:13.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-05-27T06:45:13.061 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f83e0157e40 con 0x7f83e00c7750 2024-05-27T06:45:13.064 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:13.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7f83d6000640 1 -- 172.21.15.81:0/3187765771 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f83d0016150 con 0x7f83e00c7750 2024-05-27T06:45:13.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc53406a560 con 0x7fc53415dc30 2024-05-27T06:45:13.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7fc52aa00640 1 -- 172.21.15.81:0/3682802569 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc51c05f450 con 0x7fc53415dc30 2024-05-27T06:45:13.066 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:13.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0e300ccd40 con 0x7f0e300744a0 2024-05-27T06:45:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.070+0000 7f0e26000640 1 -- 172.21.15.81:0/256336383 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0e1c05fc10 con 0x7f0e300744a0 2024-05-27T06:45:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 msgr2=0x7f83c00719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 0x7f83c00719e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f83e014cb00 tx=0x7f83cc009040 comp rx=0 tx=0).stop 2024-05-27T06:45:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 msgr2=0x7f83e0153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83d00190e0 tx=0x7f83d001ad10 comp rx=0 tx=0).stop 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 shutdown_connections 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83e0152b90 0x7f83e013a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f83c006f4d0 0x7f83c00719e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83e015c930 0x7f83e013a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 --2- 172.21.15.81:0/3187765771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83e00c7750 0x7f83e0153ad0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 >> 172.21.15.81:0/3187765771 conn(0x7f83e006cd90 msgr2=0x7f83e013ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.068 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 shutdown_connections 2024-05-27T06:45:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f83df200640 1 -- 172.21.15.81:0/3187765771 wait complete. 2024-05-27T06:45:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 msgr2=0x7fc510071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 0x7fc510071990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc5340c4ad0 tx=0x7fc524008040 comp rx=0 tx=0).stop 2024-05-27T06:45:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 msgr2=0x7fc53414a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc51c00c420 tx=0x7fc51c00c8e0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 shutdown_connections 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc53415e040 0x7fc53414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc51006f480 0x7fc510071990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc53415dc30 0x7fc53414a1b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 --2- 172.21.15.81:0/3682802569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc534154750 0x7fc534153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 >> 172.21.15.81:0/3682802569 conn(0x7fc53406cd90 msgr2=0x7fc5340c8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 shutdown_connections 2024-05-27T06:45:13.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7fc539400640 1 -- 172.21.15.81:0/3682802569 wait complete. 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 msgr2=0x7f0e10071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 0x7f0e10071960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0e2801f150 tx=0x7f0e280aa000 comp rx=0 tx=0).stop 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 msgr2=0x7f0e300ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.074+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00e8a0 tx=0x7f0e1c00ed60 comp rx=0 tx=0).stop 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 shutdown_connections 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0e300744a0 0x7f0e300ce2c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0e1006f450 0x7f0e10071960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e301545f0 0x7f0e30143430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 --2- 172.21.15.81:0/256336383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e30071f90 0x7f0e3014fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 >> 172.21.15.81:0/256336383 conn(0x7f0e30068ab0 msgr2=0x7f0e300cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 shutdown_connections 2024-05-27T06:45:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.078+0000 7f0e2f200640 1 -- 172.21.15.81:0/256336383 wait complete. 2024-05-27T06:45:13.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.082+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff1ec005860 con 0x7ff2140c9310 2024-05-27T06:45:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.086+0000 7ff210c00640 1 -- 172.21.15.81:0/31653933 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff1fc05c610 con 0x7ff2140c9310 2024-05-27T06:45:13.079 INFO:teuthology.orchestra.run.smithi081.stdout:55834574855 2024-05-27T06:45:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.086+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 msgr2=0x7ff1f00719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.086+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 0x7ff1f00719e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff204001050 tx=0x7ff204004c80 comp rx=0 tx=0).stop 2024-05-27T06:45:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.086+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 msgr2=0x7ff21414eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.086+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00b410 tx=0x7ff1fc00b8d0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 shutdown_connections 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff1f006f4d0 0x7ff1f00719e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff21415bf60 0x7ff214151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2140c9310 0x7ff21414eb90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 --2- 172.21.15.81:0/31653933 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff21406f910 0x7ff21415a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.090+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 >> 172.21.15.81:0/31653933 conn(0x7ff21406d150 msgr2=0x7ff214157b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.094+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 shutdown_connections 2024-05-27T06:45:13.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.094+0000 7ff209600640 1 -- 172.21.15.81:0/31653933 wait complete. 2024-05-27T06:45:13.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-05-27T06:45:13.089 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2024-05-27T06:45:13.096 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-05-27T06:45:13.101 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.108 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-05-27T06:45:13.109 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2024-05-27T06:45:13.112 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.118+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f443c156300 con 0x7f443c152c80 2024-05-27T06:45:13.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.118+0000 7f4432a00640 1 -- 172.21.15.81:0/2631937994 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f442805b490 con 0x7f443c152c80 2024-05-27T06:45:13.114 INFO:teuthology.orchestra.run.smithi081.stdout:55834574854 2024-05-27T06:45:13.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 msgr2=0x7f441c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 0x7f441c071910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f44340ae000 tx=0x7f44340a7070 comp rx=0 tx=0).stop 2024-05-27T06:45:13.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 msgr2=0x7f443c1441c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f442800c3f0 tx=0x7f442800c8b0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 shutdown_connections 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f443c0c8e30 0x7f443c14e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f441c06f400 0x7f441c071910 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f443c152c80 0x7f443c1441c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 --2- 172.21.15.81:0/2631937994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f443c06fda0 0x7f443c073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 >> 172.21.15.81:0/2631937994 conn(0x7f443c06cd90 msgr2=0x7f443c150520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 shutdown_connections 2024-05-27T06:45:13.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.122+0000 7f4441800640 1 -- 172.21.15.81:0/2631937994 wait complete. 2024-05-27T06:45:13.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-05-27T06:45:13.136 DEBUG:teuthology.parallel:result is None 2024-05-27T06:45:13.136 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-27T06:45:13.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:13.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 Processor -- start 2024-05-27T06:45:13.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 -- start start 2024-05-27T06:45:13.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7ee01546f0 0x7f7ee0072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee015db00 con 0x7f7ee00732f0 2024-05-27T06:45:13.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee0073860 con 0x7f7ee015a510 2024-05-27T06:45:13.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.330+0000 7f7ee5a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7ee015e6f0 con 0x7f7ee01546f0 2024-05-27T06:45:13.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ed6000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7ee00732f0 0x7f7ee00c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37184/0 (socket says 172.21.15.81:37184) 2024-05-27T06:45:13.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ed6000640 1 -- 172.21.15.81:0/3164086903 learned_addr learned my addr 172.21.15.81:0/3164086903 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edea00640 1 --2- 172.21.15.81:0/3164086903 >> v2:172.21.15.106:3300/0 conn(0x7f7ee01546f0 0x7f7ee0072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273473418 0 0) 0x7f7ee015db00 con 0x7f7ee00732f0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ec4003530 con 0x7f7ee00732f0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1083871695 0 0) 0x7f7ec4003530 con 0x7f7ee00732f0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 >> v2:172.21.15.106:3300/0 conn(0x7f7ee01546f0 msgr2=0x7f7ee0072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 --2- 172.21.15.81:0/3164086903 >> v2:172.21.15.106:3300/0 conn(0x7f7ee01546f0 0x7f7ee0072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 >> v1:172.21.15.106:6789/0 conn(0x7f7ee015a510 legacy=0x7f7ee015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee015ddc0 con 0x7f7ee00732f0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f7ec8001700 con 0x7f7ee00732f0 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 >> v1:172.21.15.81:6789/0 conn(0x7f7ee00732f0 legacy=0x7f7ee00c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec4003a10 0x7f7ec4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ec4004810 0x7f7ec4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 0x7f7ec400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee015e6f0 con 0x7f7ec40051d0 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec400c0d0 con 0x7f7ec4003a10 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec400c2d0 con 0x7f7ec4004810 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 0x7f7ec400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edea00640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec4003a10 0x7f7ec4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7ed6000640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ec4004810 0x7f7ec4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 -- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ec4004810 msgr2=0x7f7ec4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ec4004810 0x7f7ec4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 -- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec4003a10 msgr2=0x7f7ec4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec4003a10 0x7f7ec4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.334+0000 7f7edf400640 1 -- 172.21.15.81:0/3164086903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec4003530 con 0x7f7ec40051d0 2024-05-27T06:45:13.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7edf400640 1 -- 172.21.15.81:0/3164086903 reap_dead start 2024-05-27T06:45:13.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7edf400640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 0x7f7ec400b9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0074ec0 tx=0x7f7ed801c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a12c537845ac1b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80014e0 con 0x7f7ec40051d0 2024-05-27T06:45:13.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed80ab420 con 0x7f7ec40051d0 2024-05-27T06:45:13.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ede000640 1 -- 172.21.15.81:0/3164086903 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80ab810 con 0x7f7ec40051d0 2024-05-27T06:45:13.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 -- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 msgr2=0x7f7ec400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 0x7f7ec400b9d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0074ec0 tx=0x7f7ed801c4e0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 -- 172.21.15.81:0/3164086903 shutdown_connections 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/3164086903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ec40051d0 0x7f7ec400b9d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 -- 172.21.15.81:0/3164086903 >> 172.21.15.81:0/3164086903 conn(0x7f7ee0070700 msgr2=0x7f7ee00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 -- 172.21.15.81:0/3164086903 shutdown_connections 2024-05-27T06:45:13.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.338+0000 7f7ee5a00640 1 -- 172.21.15.81:0/3164086903 wait complete. 2024-05-27T06:45:13.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 Processor -- start 2024-05-27T06:45:13.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- start start 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec40051d0 0x7f7ee0157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ee00cc710 0x7f7ee0157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee00755b0 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee015e380 con 0x7f7ec40051d0 2024-05-27T06:45:13.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee0154b00 con 0x7f7ee00cc710 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57592/0 (socket says 172.21.15.81:57592) 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 -- 172.21.15.81:0/1036679058 learned_addr learned my addr 172.21.15.81:0/1036679058 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edea00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec40051d0 0x7f7ee0157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ed6000640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ee00cc710 0x7f7ee0157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 -- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ee00cc710 msgr2=0x7f7ee0157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ee00cc710 0x7f7ee0157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 -- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec40051d0 msgr2=0x7f7ee0157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec40051d0 0x7f7ee0157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 -- 172.21.15.81:0/1036679058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee01f5f20 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7edf400640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ed801c310 tx=0x7f7ec4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8007020 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed80071c0 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee01f6200 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8007590 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee01f67c0 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.342+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee006a560 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.346+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7ed80bd060 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.346+0000 7f7ed7400640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 0x7f7ebc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.346+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7ed81282a0 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.346+0000 7f7edea00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 0x7f7ebc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.346+0000 7f7edea00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 0x7f7ebc071910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0014450 tx=0x7f7ed000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.350+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7ed80161f0 con 0x7f7ee00cd6a0 2024-05-27T06:45:13.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.506+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7ee00c8ad0 con 0x7f7ebc06f400 2024-05-27T06:45:13.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ed7400640 1 -- 172.21.15.81:0/1036679058 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251902 (secure 0 0 0) 0x7f7ee00c8ad0 con 0x7f7ebc06f400 2024-05-27T06:45:13.509 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:13.511 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 msgr2=0x7f7ebc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 0x7f7ebc071910 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0014450 tx=0x7f7ed000b040 comp rx=0 tx=0).stop 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 msgr2=0x7f7ee0152080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ed801c310 tx=0x7f7ec4005840 comp rx=0 tx=0).stop 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 shutdown_connections 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ee00cc710 0x7f7ee0157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7ebc06f400 0x7f7ebc071910 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ee00cd6a0 0x7f7ee0152080 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 --2- 172.21.15.81:0/1036679058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ec40051d0 0x7f7ee0157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 >> 172.21.15.81:0/1036679058 conn(0x7f7ee0070700 msgr2=0x7f7ee0158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 shutdown_connections 2024-05-27T06:45:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.514+0000 7f7ee5a00640 1 -- 172.21.15.81:0/1036679058 wait complete. 2024-05-27T06:45:13.530 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-05-27T06:45:13.492959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002562"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620285+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:03.620285+0000","last_peered":"2024-05-27T06:45:03.620285+0000","last_clean":"2024-05-27T06:45:03.620285+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:03.620285+0000","last_undegraded":"2024-05-27T06:45:03.620285+0000","last_fullsized":"2024-05-27T06:45:03.620285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617672+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:03.617672+0000","last_peered":"2024-05-27T06:45:03.617672+0000","last_clean":"2024-05-27T06:45:03.617672+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:03.617672+0000","last_undegraded":"2024-05-27T06:45:03.617672+0000","last_fullsized":"2024-05-27T06:45:03.617672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542582+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:04.542582+0000","last_peered":"2024-05-27T06:45:04.542582+0000","last_clean":"2024-05-27T06:45:04.542582+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:04.542582+0000","last_undegraded":"2024-05-27T06:45:04.542582+0000","last_fullsized":"2024-05-27T06:45:04.542582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619261+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:03.619261+0000","last_peered":"2024-05-27T06:45:03.619261+0000","last_clean":"2024-05-27T06:45:03.619261+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:03.619261+0000","last_undegraded":"2024-05-27T06:45:03.619261+0000","last_fullsized":"2024-05-27T06:45:03.619261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620460+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:03.620460+0000","last_peered":"2024-05-27T06:45:03.620460+0000","last_clean":"2024-05-27T06:45:03.620460+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:03.620460+0000","last_undegraded":"2024-05-27T06:45:03.620460+0000","last_fullsized":"2024-05-27T06:45:03.620460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619485+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:03.619485+0000","last_peered":"2024-05-27T06:45:03.619485+0000","last_clean":"2024-05-27T06:45:03.619485+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:03.619485+0000","last_undegraded":"2024-05-27T06:45:03.619485+0000","last_fullsized":"2024-05-27T06:45:03.619485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618534+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:03.618534+0000","last_peered":"2024-05-27T06:45:03.618534+0000","last_clean":"2024-05-27T06:45:03.618534+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:03.618534+0000","last_undegraded":"2024-05-27T06:45:03.618534+0000","last_fullsized":"2024-05-27T06:45:03.618534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542255+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:04.542255+0000","last_peered":"2024-05-27T06:45:04.542255+0000","last_clean":"2024-05-27T06:45:04.542255+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:04.542255+0000","last_undegraded":"2024-05-27T06:45:04.542255+0000","last_fullsized":"2024-05-27T06:45:04.542255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620365+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:03.620365+0000","last_peered":"2024-05-27T06:45:03.620365+0000","last_clean":"2024-05-27T06:45:03.620365+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:03.620365+0000","last_undegraded":"2024-05-27T06:45:03.620365+0000","last_fullsized":"2024-05-27T06:45:03.620365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542559+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:04.542559+0000","last_peered":"2024-05-27T06:45:04.542559+0000","last_clean":"2024-05-27T06:45:04.542559+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:04.542559+0000","last_undegraded":"2024-05-27T06:45:04.542559+0000","last_fullsized":"2024-05-27T06:45:04.542559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542270+0000","last_change":"2024-05-27T06:45:02.647403+0000","last_active":"2024-05-27T06:45:04.542270+0000","last_peered":"2024-05-27T06:45:04.542270+0000","last_clean":"2024-05-27T06:45:04.542270+0000","last_became_active":"2024-05-27T06:45:02.647061+0000","last_became_peered":"2024-05-27T06:45:02.647061+0000","last_unstale":"2024-05-27T06:45:04.542270+0000","last_undegraded":"2024-05-27T06:45:04.542270+0000","last_fullsized":"2024-05-27T06:45:04.542270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:10:58.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543063+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:04.543063+0000","last_peered":"2024-05-27T06:45:04.543063+0000","last_clean":"2024-05-27T06:45:04.543063+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:04.543063+0000","last_undegraded":"2024-05-27T06:45:04.543063+0000","last_fullsized":"2024-05-27T06:45:04.543063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542091+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:04.542091+0000","last_peered":"2024-05-27T06:45:04.542091+0000","last_clean":"2024-05-27T06:45:04.542091+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:04.542091+0000","last_undegraded":"2024-05-27T06:45:04.542091+0000","last_fullsized":"2024-05-27T06:45:04.542091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542474+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:04.542474+0000","last_peered":"2024-05-27T06:45:04.542474+0000","last_clean":"2024-05-27T06:45:04.542474+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:04.542474+0000","last_undegraded":"2024-05-27T06:45:04.542474+0000","last_fullsized":"2024-05-27T06:45:04.542474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617212+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:03.617212+0000","last_peered":"2024-05-27T06:45:03.617212+0000","last_clean":"2024-05-27T06:45:03.617212+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:03.617212+0000","last_undegraded":"2024-05-27T06:45:03.617212+0000","last_fullsized":"2024-05-27T06:45:03.617212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619654+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:03.619654+0000","last_peered":"2024-05-27T06:45:03.619654+0000","last_clean":"2024-05-27T06:45:03.619654+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:03.619654+0000","last_undegraded":"2024-05-27T06:45:03.619654+0000","last_fullsized":"2024-05-27T06:45:03.619654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620222+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:03.620222+0000","last_peered":"2024-05-27T06:45:03.620222+0000","last_clean":"2024-05-27T06:45:03.620222+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:03.620222+0000","last_undegraded":"2024-05-27T06:45:03.620222+0000","last_fullsized":"2024-05-27T06:45:03.620222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541770+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:04.541770+0000","last_peered":"2024-05-27T06:45:04.541770+0000","last_clean":"2024-05-27T06:45:04.541770+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:04.541770+0000","last_undegraded":"2024-05-27T06:45:04.541770+0000","last_fullsized":"2024-05-27T06:45:04.541770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620170+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:03.620170+0000","last_peered":"2024-05-27T06:45:03.620170+0000","last_clean":"2024-05-27T06:45:03.620170+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:03.620170+0000","last_undegraded":"2024-05-27T06:45:03.620170+0000","last_fullsized":"2024-05-27T06:45:03.620170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542157+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:04.542157+0000","last_peered":"2024-05-27T06:45:04.542157+0000","last_clean":"2024-05-27T06:45:04.542157+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:04.542157+0000","last_undegraded":"2024-05-27T06:45:04.542157+0000","last_fullsized":"2024-05-27T06:45:04.542157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542660+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:04.542660+0000","last_peered":"2024-05-27T06:45:04.542660+0000","last_clean":"2024-05-27T06:45:04.542660+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:04.542660+0000","last_undegraded":"2024-05-27T06:45:04.542660+0000","last_fullsized":"2024-05-27T06:45:04.542660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619623+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:03.619623+0000","last_peered":"2024-05-27T06:45:03.619623+0000","last_clean":"2024-05-27T06:45:03.619623+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:03.619623+0000","last_undegraded":"2024-05-27T06:45:03.619623+0000","last_fullsized":"2024-05-27T06:45:03.619623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542082+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:04.542082+0000","last_peered":"2024-05-27T06:45:04.542082+0000","last_clean":"2024-05-27T06:45:04.542082+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:04.542082+0000","last_undegraded":"2024-05-27T06:45:04.542082+0000","last_fullsized":"2024-05-27T06:45:04.542082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619147+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:03.619147+0000","last_peered":"2024-05-27T06:45:03.619147+0000","last_clean":"2024-05-27T06:45:03.619147+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:03.619147+0000","last_undegraded":"2024-05-27T06:45:03.619147+0000","last_fullsized":"2024-05-27T06:45:03.619147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543112+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:04.543112+0000","last_peered":"2024-05-27T06:45:04.543112+0000","last_clean":"2024-05-27T06:45:04.543112+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:04.543112+0000","last_undegraded":"2024-05-27T06:45:04.543112+0000","last_fullsized":"2024-05-27T06:45:04.543112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542613+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:04.542613+0000","last_peered":"2024-05-27T06:45:04.542613+0000","last_clean":"2024-05-27T06:45:04.542613+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:04.542613+0000","last_undegraded":"2024-05-27T06:45:04.542613+0000","last_fullsized":"2024-05-27T06:45:04.542613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617693+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:03.617693+0000","last_peered":"2024-05-27T06:45:03.617693+0000","last_clean":"2024-05-27T06:45:03.617693+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:03.617693+0000","last_undegraded":"2024-05-27T06:45:03.617693+0000","last_fullsized":"2024-05-27T06:45:03.617693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541962+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:04.541962+0000","last_peered":"2024-05-27T06:45:04.541962+0000","last_clean":"2024-05-27T06:45:04.541962+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:04.541962+0000","last_undegraded":"2024-05-27T06:45:04.541962+0000","last_fullsized":"2024-05-27T06:45:04.541962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542858+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:04.542858+0000","last_peered":"2024-05-27T06:45:04.542858+0000","last_clean":"2024-05-27T06:45:04.542858+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:04.542858+0000","last_undegraded":"2024-05-27T06:45:04.542858+0000","last_fullsized":"2024-05-27T06:45:04.542858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619714+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:03.619714+0000","last_peered":"2024-05-27T06:45:03.619714+0000","last_clean":"2024-05-27T06:45:03.619714+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:03.619714+0000","last_undegraded":"2024-05-27T06:45:03.619714+0000","last_fullsized":"2024-05-27T06:45:03.619714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542030+0000","last_change":"2024-05-27T06:45:02.627394+0000","last_active":"2024-05-27T06:45:04.542030+0000","last_peered":"2024-05-27T06:45:04.542030+0000","last_clean":"2024-05-27T06:45:04.542030+0000","last_became_active":"2024-05-27T06:45:02.626550+0000","last_became_peered":"2024-05-27T06:45:02.626550+0000","last_unstale":"2024-05-27T06:45:04.542030+0000","last_undegraded":"2024-05-27T06:45:04.542030+0000","last_fullsized":"2024-05-27T06:45:04.542030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:01.383493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541845+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:04.541845+0000","last_peered":"2024-05-27T06:45:04.541845+0000","last_clean":"2024-05-27T06:45:04.541845+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:04.541845+0000","last_undegraded":"2024-05-27T06:45:04.541845+0000","last_fullsized":"2024-05-27T06:45:04.541845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542588+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:04.542588+0000","last_peered":"2024-05-27T06:45:04.542588+0000","last_clean":"2024-05-27T06:45:04.542588+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:04.542588+0000","last_undegraded":"2024-05-27T06:45:04.542588+0000","last_fullsized":"2024-05-27T06:45:04.542588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542786+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:04.542786+0000","last_peered":"2024-05-27T06:45:04.542786+0000","last_clean":"2024-05-27T06:45:04.542786+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:04.542786+0000","last_undegraded":"2024-05-27T06:45:04.542786+0000","last_fullsized":"2024-05-27T06:45:04.542786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543053+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:04.543053+0000","last_peered":"2024-05-27T06:45:04.543053+0000","last_clean":"2024-05-27T06:45:04.543053+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:04.543053+0000","last_undegraded":"2024-05-27T06:45:04.543053+0000","last_fullsized":"2024-05-27T06:45:04.543053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541946+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:04.541946+0000","last_peered":"2024-05-27T06:45:04.541946+0000","last_clean":"2024-05-27T06:45:04.541946+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:04.541946+0000","last_undegraded":"2024-05-27T06:45:04.541946+0000","last_fullsized":"2024-05-27T06:45:04.541946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541609+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:04.541609+0000","last_peered":"2024-05-27T06:45:04.541609+0000","last_clean":"2024-05-27T06:45:04.541609+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:04.541609+0000","last_undegraded":"2024-05-27T06:45:04.541609+0000","last_fullsized":"2024-05-27T06:45:04.541609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542424+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:04.542424+0000","last_peered":"2024-05-27T06:45:04.542424+0000","last_clean":"2024-05-27T06:45:04.542424+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:04.542424+0000","last_undegraded":"2024-05-27T06:45:04.542424+0000","last_fullsized":"2024-05-27T06:45:04.542424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619231+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:03.619231+0000","last_peered":"2024-05-27T06:45:03.619231+0000","last_clean":"2024-05-27T06:45:03.619231+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:03.619231+0000","last_undegraded":"2024-05-27T06:45:03.619231+0000","last_fullsized":"2024-05-27T06:45:03.619231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617259+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:03.617259+0000","last_peered":"2024-05-27T06:45:03.617259+0000","last_clean":"2024-05-27T06:45:03.617259+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:03.617259+0000","last_undegraded":"2024-05-27T06:45:03.617259+0000","last_fullsized":"2024-05-27T06:45:03.617259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542316+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:04.542316+0000","last_peered":"2024-05-27T06:45:04.542316+0000","last_clean":"2024-05-27T06:45:04.542316+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:04.542316+0000","last_undegraded":"2024-05-27T06:45:04.542316+0000","last_fullsized":"2024-05-27T06:45:04.542316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619378+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:03.619378+0000","last_peered":"2024-05-27T06:45:03.619378+0000","last_clean":"2024-05-27T06:45:03.619378+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:03.619378+0000","last_undegraded":"2024-05-27T06:45:03.619378+0000","last_fullsized":"2024-05-27T06:45:03.619378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619917+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:03.619917+0000","last_peered":"2024-05-27T06:45:03.619917+0000","last_clean":"2024-05-27T06:45:03.619917+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:03.619917+0000","last_undegraded":"2024-05-27T06:45:03.619917+0000","last_fullsized":"2024-05-27T06:45:03.619917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543182+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:04.543182+0000","last_peered":"2024-05-27T06:45:04.543182+0000","last_clean":"2024-05-27T06:45:04.543182+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:04.543182+0000","last_undegraded":"2024-05-27T06:45:04.543182+0000","last_fullsized":"2024-05-27T06:45:04.543182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641647+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:07.641647+0000","last_peered":"2024-05-27T06:45:07.641647+0000","last_clean":"2024-05-27T06:45:07.641647+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:07.641647+0000","last_undegraded":"2024-05-27T06:45:07.641647+0000","last_fullsized":"2024-05-27T06:45:07.641647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543150+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:04.543150+0000","last_peered":"2024-05-27T06:45:04.543150+0000","last_clean":"2024-05-27T06:45:04.543150+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:04.543150+0000","last_undegraded":"2024-05-27T06:45:04.543150+0000","last_fullsized":"2024-05-27T06:45:04.543150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.643949+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:07.643949+0000","last_peered":"2024-05-27T06:45:07.643949+0000","last_clean":"2024-05-27T06:45:07.643949+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:07.643949+0000","last_undegraded":"2024-05-27T06:45:07.643949+0000","last_fullsized":"2024-05-27T06:45:07.643949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618838+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:03.618838+0000","last_peered":"2024-05-27T06:45:03.618838+0000","last_clean":"2024-05-27T06:45:03.618838+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:03.618838+0000","last_undegraded":"2024-05-27T06:45:03.618838+0000","last_fullsized":"2024-05-27T06:45:03.618838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641947+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:07.641947+0000","last_peered":"2024-05-27T06:45:07.641947+0000","last_clean":"2024-05-27T06:45:07.641947+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:07.641947+0000","last_undegraded":"2024-05-27T06:45:07.641947+0000","last_fullsized":"2024-05-27T06:45:07.641947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542669+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:04.542669+0000","last_peered":"2024-05-27T06:45:04.542669+0000","last_clean":"2024-05-27T06:45:04.542669+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:04.542669+0000","last_undegraded":"2024-05-27T06:45:04.542669+0000","last_fullsized":"2024-05-27T06:45:04.542669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619690+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.619690+0000","last_peered":"2024-05-27T06:45:03.619690+0000","last_clean":"2024-05-27T06:45:03.619690+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:03.619690+0000","last_undegraded":"2024-05-27T06:45:03.619690+0000","last_fullsized":"2024-05-27T06:45:03.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619590+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:03.619590+0000","last_peered":"2024-05-27T06:45:03.619590+0000","last_clean":"2024-05-27T06:45:03.619590+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:03.619590+0000","last_undegraded":"2024-05-27T06:45:03.619590+0000","last_fullsized":"2024-05-27T06:45:03.619590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541708+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:04.541708+0000","last_peered":"2024-05-27T06:45:04.541708+0000","last_clean":"2024-05-27T06:45:04.541708+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:04.541708+0000","last_undegraded":"2024-05-27T06:45:04.541708+0000","last_fullsized":"2024-05-27T06:45:04.541708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541996+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:04.541996+0000","last_peered":"2024-05-27T06:45:04.541996+0000","last_clean":"2024-05-27T06:45:04.541996+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:04.541996+0000","last_undegraded":"2024-05-27T06:45:04.541996+0000","last_fullsized":"2024-05-27T06:45:04.541996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.643262+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:07.643262+0000","last_peered":"2024-05-27T06:45:07.643262+0000","last_clean":"2024-05-27T06:45:07.643262+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:07.643262+0000","last_undegraded":"2024-05-27T06:45:07.643262+0000","last_fullsized":"2024-05-27T06:45:07.643262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619502+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:03.619502+0000","last_peered":"2024-05-27T06:45:03.619502+0000","last_clean":"2024-05-27T06:45:03.619502+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:03.619502+0000","last_undegraded":"2024-05-27T06:45:03.619502+0000","last_fullsized":"2024-05-27T06:45:03.619502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543172+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:04.543172+0000","last_peered":"2024-05-27T06:45:04.543172+0000","last_clean":"2024-05-27T06:45:04.543172+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:04.543172+0000","last_undegraded":"2024-05-27T06:45:04.543172+0000","last_fullsized":"2024-05-27T06:45:04.543172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620404+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:03.620404+0000","last_peered":"2024-05-27T06:45:03.620404+0000","last_clean":"2024-05-27T06:45:03.620404+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:03.620404+0000","last_undegraded":"2024-05-27T06:45:03.620404+0000","last_fullsized":"2024-05-27T06:45:03.620404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641718+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:07.641718+0000","last_peered":"2024-05-27T06:45:07.641718+0000","last_clean":"2024-05-27T06:45:07.641718+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:07.641718+0000","last_undegraded":"2024-05-27T06:45:07.641718+0000","last_fullsized":"2024-05-27T06:45:07.641718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620405+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:03.620405+0000","last_peered":"2024-05-27T06:45:03.620405+0000","last_clean":"2024-05-27T06:45:03.620405+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:03.620405+0000","last_undegraded":"2024-05-27T06:45:03.620405+0000","last_fullsized":"2024-05-27T06:45:03.620405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617687+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:03.617687+0000","last_peered":"2024-05-27T06:45:03.617687+0000","last_clean":"2024-05-27T06:45:03.617687+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:03.617687+0000","last_undegraded":"2024-05-27T06:45:03.617687+0000","last_fullsized":"2024-05-27T06:45:03.617687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619736+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:03.619736+0000","last_peered":"2024-05-27T06:45:03.619736+0000","last_clean":"2024-05-27T06:45:03.619736+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:03.619736+0000","last_undegraded":"2024-05-27T06:45:03.619736+0000","last_fullsized":"2024-05-27T06:45:03.619736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541528+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:04.541528+0000","last_peered":"2024-05-27T06:45:04.541528+0000","last_clean":"2024-05-27T06:45:04.541528+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:04.541528+0000","last_undegraded":"2024-05-27T06:45:04.541528+0000","last_fullsized":"2024-05-27T06:45:04.541528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620105+0000","last_change":"2024-05-27T06:45:02.651256+0000","last_active":"2024-05-27T06:45:03.620105+0000","last_peered":"2024-05-27T06:45:03.620105+0000","last_clean":"2024-05-27T06:45:03.620105+0000","last_became_active":"2024-05-27T06:45:02.650646+0000","last_became_peered":"2024-05-27T06:45:02.650646+0000","last_unstale":"2024-05-27T06:45:03.620105+0000","last_undegraded":"2024-05-27T06:45:03.620105+0000","last_fullsized":"2024-05-27T06:45:03.620105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:39:57.469688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618689+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:03.618689+0000","last_peered":"2024-05-27T06:45:03.618689+0000","last_clean":"2024-05-27T06:45:03.618689+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:03.618689+0000","last_undegraded":"2024-05-27T06:45:03.618689+0000","last_fullsized":"2024-05-27T06:45:03.618689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619124+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:03.619124+0000","last_peered":"2024-05-27T06:45:03.619124+0000","last_clean":"2024-05-27T06:45:03.619124+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:03.619124+0000","last_undegraded":"2024-05-27T06:45:03.619124+0000","last_fullsized":"2024-05-27T06:45:03.619124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541730+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:04.541730+0000","last_peered":"2024-05-27T06:45:04.541730+0000","last_clean":"2024-05-27T06:45:04.541730+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:04.541730+0000","last_undegraded":"2024-05-27T06:45:04.541730+0000","last_fullsized":"2024-05-27T06:45:04.541730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618565+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:03.618565+0000","last_peered":"2024-05-27T06:45:03.618565+0000","last_clean":"2024-05-27T06:45:03.618565+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:03.618565+0000","last_undegraded":"2024-05-27T06:45:03.618565+0000","last_fullsized":"2024-05-27T06:45:03.618565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543183+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:04.543183+0000","last_peered":"2024-05-27T06:45:04.543183+0000","last_clean":"2024-05-27T06:45:04.543183+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:04.543183+0000","last_undegraded":"2024-05-27T06:45:04.543183+0000","last_fullsized":"2024-05-27T06:45:04.543183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542094+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:04.542094+0000","last_peered":"2024-05-27T06:45:04.542094+0000","last_clean":"2024-05-27T06:45:04.542094+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:04.542094+0000","last_undegraded":"2024-05-27T06:45:04.542094+0000","last_fullsized":"2024-05-27T06:45:04.542094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617210+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:03.617210+0000","last_peered":"2024-05-27T06:45:03.617210+0000","last_clean":"2024-05-27T06:45:03.617210+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:03.617210+0000","last_undegraded":"2024-05-27T06:45:03.617210+0000","last_fullsized":"2024-05-27T06:45:03.617210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619757+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:03.619757+0000","last_peered":"2024-05-27T06:45:03.619757+0000","last_clean":"2024-05-27T06:45:03.619757+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:03.619757+0000","last_undegraded":"2024-05-27T06:45:03.619757+0000","last_fullsized":"2024-05-27T06:45:03.619757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542806+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:04.542806+0000","last_peered":"2024-05-27T06:45:04.542806+0000","last_clean":"2024-05-27T06:45:04.542806+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:04.542806+0000","last_undegraded":"2024-05-27T06:45:04.542806+0000","last_fullsized":"2024-05-27T06:45:04.542806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617419+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:03.617419+0000","last_peered":"2024-05-27T06:45:03.617419+0000","last_clean":"2024-05-27T06:45:03.617419+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:03.617419+0000","last_undegraded":"2024-05-27T06:45:03.617419+0000","last_fullsized":"2024-05-27T06:45:03.617419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619493+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:03.619493+0000","last_peered":"2024-05-27T06:45:03.619493+0000","last_clean":"2024-05-27T06:45:03.619493+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:03.619493+0000","last_undegraded":"2024-05-27T06:45:03.619493+0000","last_fullsized":"2024-05-27T06:45:03.619493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620147+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.620147+0000","last_peered":"2024-05-27T06:45:03.620147+0000","last_clean":"2024-05-27T06:45:03.620147+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:03.620147+0000","last_undegraded":"2024-05-27T06:45:03.620147+0000","last_fullsized":"2024-05-27T06:45:03.620147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617526+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:03.617526+0000","last_peered":"2024-05-27T06:45:03.617526+0000","last_clean":"2024-05-27T06:45:03.617526+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:03.617526+0000","last_undegraded":"2024-05-27T06:45:03.617526+0000","last_fullsized":"2024-05-27T06:45:03.617526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617705+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:03.617705+0000","last_peered":"2024-05-27T06:45:03.617705+0000","last_clean":"2024-05-27T06:45:03.617705+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:03.617705+0000","last_undegraded":"2024-05-27T06:45:03.617705+0000","last_fullsized":"2024-05-27T06:45:03.617705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543116+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:04.543116+0000","last_peered":"2024-05-27T06:45:04.543116+0000","last_clean":"2024-05-27T06:45:04.543116+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:04.543116+0000","last_undegraded":"2024-05-27T06:45:04.543116+0000","last_fullsized":"2024-05-27T06:45:04.543116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641734+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:07.641734+0000","last_peered":"2024-05-27T06:45:07.641734+0000","last_clean":"2024-05-27T06:45:07.641734+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:07.641734+0000","last_undegraded":"2024-05-27T06:45:07.641734+0000","last_fullsized":"2024-05-27T06:45:07.641734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542819+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:04.542819+0000","last_peered":"2024-05-27T06:45:04.542819+0000","last_clean":"2024-05-27T06:45:04.542819+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:04.542819+0000","last_undegraded":"2024-05-27T06:45:04.542819+0000","last_fullsized":"2024-05-27T06:45:04.542819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618413+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:03.618413+0000","last_peered":"2024-05-27T06:45:03.618413+0000","last_clean":"2024-05-27T06:45:03.618413+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:03.618413+0000","last_undegraded":"2024-05-27T06:45:03.618413+0000","last_fullsized":"2024-05-27T06:45:03.618413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542399+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:04.542399+0000","last_peered":"2024-05-27T06:45:04.542399+0000","last_clean":"2024-05-27T06:45:04.542399+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:04.542399+0000","last_undegraded":"2024-05-27T06:45:04.542399+0000","last_fullsized":"2024-05-27T06:45:04.542399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617281+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:03.617281+0000","last_peered":"2024-05-27T06:45:03.617281+0000","last_clean":"2024-05-27T06:45:03.617281+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:03.617281+0000","last_undegraded":"2024-05-27T06:45:03.617281+0000","last_fullsized":"2024-05-27T06:45:03.617281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542603+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:04.542603+0000","last_peered":"2024-05-27T06:45:04.542603+0000","last_clean":"2024-05-27T06:45:04.542603+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:04.542603+0000","last_undegraded":"2024-05-27T06:45:04.542603+0000","last_fullsized":"2024-05-27T06:45:04.542603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617374+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:03.617374+0000","last_peered":"2024-05-27T06:45:03.617374+0000","last_clean":"2024-05-27T06:45:03.617374+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:03.617374+0000","last_undegraded":"2024-05-27T06:45:03.617374+0000","last_fullsized":"2024-05-27T06:45:03.617374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543008+0000","last_change":"2024-05-27T06:45:02.632527+0000","last_active":"2024-05-27T06:45:04.543008+0000","last_peered":"2024-05-27T06:45:04.543008+0000","last_clean":"2024-05-27T06:45:04.543008+0000","last_became_active":"2024-05-27T06:45:02.632140+0000","last_became_peered":"2024-05-27T06:45:02.632140+0000","last_unstale":"2024-05-27T06:45:04.543008+0000","last_undegraded":"2024-05-27T06:45:04.543008+0000","last_fullsized":"2024-05-27T06:45:04.543008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.644734+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:07.644734+0000","last_peered":"2024-05-27T06:45:07.644734+0000","last_clean":"2024-05-27T06:45:07.644734+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:07.644734+0000","last_undegraded":"2024-05-27T06:45:07.644734+0000","last_fullsized":"2024-05-27T06:45:07.644734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541876+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:04.541876+0000","last_peered":"2024-05-27T06:45:04.541876+0000","last_clean":"2024-05-27T06:45:04.541876+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:04.541876+0000","last_undegraded":"2024-05-27T06:45:04.541876+0000","last_fullsized":"2024-05-27T06:45:04.541876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619831+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:03.619831+0000","last_peered":"2024-05-27T06:45:03.619831+0000","last_clean":"2024-05-27T06:45:03.619831+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:03.619831+0000","last_undegraded":"2024-05-27T06:45:03.619831+0000","last_fullsized":"2024-05-27T06:45:03.619831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641892+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:07.641892+0000","last_peered":"2024-05-27T06:45:07.641892+0000","last_clean":"2024-05-27T06:45:07.641892+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:07.641892+0000","last_undegraded":"2024-05-27T06:45:07.641892+0000","last_fullsized":"2024-05-27T06:45:07.641892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619460+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:03.619460+0000","last_peered":"2024-05-27T06:45:03.619460+0000","last_clean":"2024-05-27T06:45:03.619460+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:03.619460+0000","last_undegraded":"2024-05-27T06:45:03.619460+0000","last_fullsized":"2024-05-27T06:45:03.619460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.642913+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:07.642913+0000","last_peered":"2024-05-27T06:45:07.642913+0000","last_clean":"2024-05-27T06:45:07.642913+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:07.642913+0000","last_undegraded":"2024-05-27T06:45:07.642913+0000","last_fullsized":"2024-05-27T06:45:07.642913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542992+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:04.542992+0000","last_peered":"2024-05-27T06:45:04.542992+0000","last_clean":"2024-05-27T06:45:04.542992+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:04.542992+0000","last_undegraded":"2024-05-27T06:45:04.542992+0000","last_fullsized":"2024-05-27T06:45:04.542992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542987+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:04.542987+0000","last_peered":"2024-05-27T06:45:04.542987+0000","last_clean":"2024-05-27T06:45:04.542987+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:04.542987+0000","last_undegraded":"2024-05-27T06:45:04.542987+0000","last_fullsized":"2024-05-27T06:45:04.542987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541662+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:04.541662+0000","last_peered":"2024-05-27T06:45:04.541662+0000","last_clean":"2024-05-27T06:45:04.541662+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:04.541662+0000","last_undegraded":"2024-05-27T06:45:04.541662+0000","last_fullsized":"2024-05-27T06:45:04.541662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.644507+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:07.644507+0000","last_peered":"2024-05-27T06:45:07.644507+0000","last_clean":"2024-05-27T06:45:07.644507+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:07.644507+0000","last_undegraded":"2024-05-27T06:45:07.644507+0000","last_fullsized":"2024-05-27T06:45:07.644507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:13.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:13.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.730+0000 7f26f2800640 1 Processor -- start 2024-05-27T06:45:13.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.730+0000 7f26f2800640 1 -- start start 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26f2800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26ec06c790 0x7f26ec06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26f2800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec0c8880 0x7f26ec0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26f2800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec0c9f80 con 0x7f26ec15cb80 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26f2800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f26ec0c58b0 con 0x7f26ec06c790 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26f2800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec0c83f0 con 0x7f26ec0c8880 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26ec06c790 0x7f26ec06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26ec06c790 0x7f26ec06cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57602/0 (socket says 172.21.15.81:57602) 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 -- 172.21.15.81:0/3171142724 learned_addr learned my addr 172.21.15.81:0/3171142724 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26ebe00640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec0c8880 0x7f26ec0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 -- 172.21.15.81:0/3171142724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec0c8880 msgr2=0x7f26ec0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec0c8880 0x7f26ec0c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 -- 172.21.15.81:0/3171142724 >> v1:172.21.15.106:6789/0 conn(0x7f26ec15cb80 legacy=0x7f26ec152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.734+0000 7f26e2a00640 1 -- 172.21.15.81:0/3171142724 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec0c52f0 con 0x7f26ec06c790 2024-05-27T06:45:13.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26e2a00640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec06c790 0x7f26ec06cba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009870 tx=0x7f26d401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce470b277b41d3ff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26eaa00640 1 -- 172.21.15.81:0/3171142724 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40b2070 con 0x7f26ec06c790 2024-05-27T06:45:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26eaa00640 1 -- 172.21.15.81:0/3171142724 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d40a9070 con 0x7f26ec06c790 2024-05-27T06:45:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26eaa00640 1 -- 172.21.15.81:0/3171142724 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40ad880 con 0x7f26ec06c790 2024-05-27T06:45:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 -- 172.21.15.81:0/3171142724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec06c790 msgr2=0x7f26ec06cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec06c790 0x7f26ec06cba0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009870 tx=0x7f26d401c560 comp rx=0 tx=0).stop 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 -- 172.21.15.81:0/3171142724 shutdown_connections 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec0c8880 0x7f26ec0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 --2- 172.21.15.81:0/3171142724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec06c790 0x7f26ec06cba0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 -- 172.21.15.81:0/3171142724 >> 172.21.15.81:0/3171142724 conn(0x7f26ec070700 msgr2=0x7f26ec0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 -- 172.21.15.81:0/3171142724 shutdown_connections 2024-05-27T06:45:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 -- 172.21.15.81:0/3171142724 wait complete. 2024-05-27T06:45:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.738+0000 7f26f2800640 1 Processor -- start 2024-05-27T06:45:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- start start 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec06c790 0x7f26ec13e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ec15cb80 0x7f26ec1f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec0c9c80 con 0x7f26ec0c8880 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec0c9b00 con 0x7f26ec15cb80 2024-05-27T06:45:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec0cdfd0 con 0x7f26ec06c790 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57616/0 (socket says 172.21.15.81:57616) 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 -- 172.21.15.81:0/2540998610 learned_addr learned my addr 172.21.15.81:0/2540998610 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26eb400640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec06c790 0x7f26ec13e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26ebe00640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ec15cb80 0x7f26ec1f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 -- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec06c790 msgr2=0x7f26ec13e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec06c790 0x7f26ec13e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 -- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ec15cb80 msgr2=0x7f26ec1f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ec15cb80 0x7f26ec1f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 -- 172.21.15.81:0/2540998610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec154db0 con 0x7f26ec0c8880 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e2a00640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b0040 tx=0x7f26d4019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40b2040 con 0x7f26ec0c8880 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d40bb070 con 0x7f26ec0c8880 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40b6440 con 0x7f26ec0c8880 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec14a1b0 con 0x7f26ec0c8880 2024-05-27T06:45:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec14a470 con 0x7f26ec0c8880 2024-05-27T06:45:13.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.742+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec06a560 con 0x7f26ec0c8880 2024-05-27T06:45:13.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.746+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f26d40ab070 con 0x7f26ec0c8880 2024-05-27T06:45:13.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.746+0000 7f26e3e00640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 0x7f26c8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:13.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.746+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f26d4128b10 con 0x7f26ec0c8880 2024-05-27T06:45:13.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.746+0000 7f26eb400640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 0x7f26c8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:13.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.746+0000 7f26eb400640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 0x7f26c8071940 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26dc007fd0 tx=0x7f26dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:13.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.750+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f26d4016150 con 0x7f26ec0c8880 2024-05-27T06:45:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.906+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26ec156340 con 0x7f26c806f430 2024-05-27T06:45:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.910+0000 7f26e3e00640 1 -- 172.21.15.81:0/2540998610 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251902 (secure 0 0 0) 0x7f26ec156340 con 0x7f26c806f430 2024-05-27T06:45:13.909 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:13.910 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 msgr2=0x7f26c8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 0x7f26c8071940 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26dc007fd0 tx=0x7f26dc009040 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 msgr2=0x7f26ec154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b0040 tx=0x7f26d4019300 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 shutdown_connections 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f26c806f430 0x7f26c8071940 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26ec15cb80 0x7f26ec1f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c8880 0x7f26ec154870 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 --2- 172.21.15.81:0/2540998610 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26ec06c790 0x7f26ec13e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 >> 172.21.15.81:0/2540998610 conn(0x7f26ec070700 msgr2=0x7f26ec0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 shutdown_connections 2024-05-27T06:45:13.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:13.914+0000 7f26f2800640 1 -- 172.21.15.81:0/2540998610 wait complete. 2024-05-27T06:45:13.930 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-05-27T06:45:13.492959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218016,"kb_used_data":1888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756704,"statfs":{"total":773094113280,"available":772870864896,"internally_reserved":0,"allocated":1933312,"data_stored":1790396,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002562"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620285+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:03.620285+0000","last_peered":"2024-05-27T06:45:03.620285+0000","last_clean":"2024-05-27T06:45:03.620285+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:03.620285+0000","last_undegraded":"2024-05-27T06:45:03.620285+0000","last_fullsized":"2024-05-27T06:45:03.620285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617672+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:03.617672+0000","last_peered":"2024-05-27T06:45:03.617672+0000","last_clean":"2024-05-27T06:45:03.617672+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:03.617672+0000","last_undegraded":"2024-05-27T06:45:03.617672+0000","last_fullsized":"2024-05-27T06:45:03.617672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542582+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:04.542582+0000","last_peered":"2024-05-27T06:45:04.542582+0000","last_clean":"2024-05-27T06:45:04.542582+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:04.542582+0000","last_undegraded":"2024-05-27T06:45:04.542582+0000","last_fullsized":"2024-05-27T06:45:04.542582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619261+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:03.619261+0000","last_peered":"2024-05-27T06:45:03.619261+0000","last_clean":"2024-05-27T06:45:03.619261+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:03.619261+0000","last_undegraded":"2024-05-27T06:45:03.619261+0000","last_fullsized":"2024-05-27T06:45:03.619261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620460+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:03.620460+0000","last_peered":"2024-05-27T06:45:03.620460+0000","last_clean":"2024-05-27T06:45:03.620460+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:03.620460+0000","last_undegraded":"2024-05-27T06:45:03.620460+0000","last_fullsized":"2024-05-27T06:45:03.620460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619485+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:03.619485+0000","last_peered":"2024-05-27T06:45:03.619485+0000","last_clean":"2024-05-27T06:45:03.619485+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:03.619485+0000","last_undegraded":"2024-05-27T06:45:03.619485+0000","last_fullsized":"2024-05-27T06:45:03.619485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618534+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:03.618534+0000","last_peered":"2024-05-27T06:45:03.618534+0000","last_clean":"2024-05-27T06:45:03.618534+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:03.618534+0000","last_undegraded":"2024-05-27T06:45:03.618534+0000","last_fullsized":"2024-05-27T06:45:03.618534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542255+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:04.542255+0000","last_peered":"2024-05-27T06:45:04.542255+0000","last_clean":"2024-05-27T06:45:04.542255+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:04.542255+0000","last_undegraded":"2024-05-27T06:45:04.542255+0000","last_fullsized":"2024-05-27T06:45:04.542255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620365+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:03.620365+0000","last_peered":"2024-05-27T06:45:03.620365+0000","last_clean":"2024-05-27T06:45:03.620365+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:03.620365+0000","last_undegraded":"2024-05-27T06:45:03.620365+0000","last_fullsized":"2024-05-27T06:45:03.620365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542559+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:04.542559+0000","last_peered":"2024-05-27T06:45:04.542559+0000","last_clean":"2024-05-27T06:45:04.542559+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:04.542559+0000","last_undegraded":"2024-05-27T06:45:04.542559+0000","last_fullsized":"2024-05-27T06:45:04.542559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542270+0000","last_change":"2024-05-27T06:45:02.647403+0000","last_active":"2024-05-27T06:45:04.542270+0000","last_peered":"2024-05-27T06:45:04.542270+0000","last_clean":"2024-05-27T06:45:04.542270+0000","last_became_active":"2024-05-27T06:45:02.647061+0000","last_became_peered":"2024-05-27T06:45:02.647061+0000","last_unstale":"2024-05-27T06:45:04.542270+0000","last_undegraded":"2024-05-27T06:45:04.542270+0000","last_fullsized":"2024-05-27T06:45:04.542270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:10:58.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543063+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:04.543063+0000","last_peered":"2024-05-27T06:45:04.543063+0000","last_clean":"2024-05-27T06:45:04.543063+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:04.543063+0000","last_undegraded":"2024-05-27T06:45:04.543063+0000","last_fullsized":"2024-05-27T06:45:04.543063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542091+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:04.542091+0000","last_peered":"2024-05-27T06:45:04.542091+0000","last_clean":"2024-05-27T06:45:04.542091+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:04.542091+0000","last_undegraded":"2024-05-27T06:45:04.542091+0000","last_fullsized":"2024-05-27T06:45:04.542091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542474+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:04.542474+0000","last_peered":"2024-05-27T06:45:04.542474+0000","last_clean":"2024-05-27T06:45:04.542474+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:04.542474+0000","last_undegraded":"2024-05-27T06:45:04.542474+0000","last_fullsized":"2024-05-27T06:45:04.542474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617212+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:03.617212+0000","last_peered":"2024-05-27T06:45:03.617212+0000","last_clean":"2024-05-27T06:45:03.617212+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:03.617212+0000","last_undegraded":"2024-05-27T06:45:03.617212+0000","last_fullsized":"2024-05-27T06:45:03.617212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619654+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:03.619654+0000","last_peered":"2024-05-27T06:45:03.619654+0000","last_clean":"2024-05-27T06:45:03.619654+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:03.619654+0000","last_undegraded":"2024-05-27T06:45:03.619654+0000","last_fullsized":"2024-05-27T06:45:03.619654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620222+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:03.620222+0000","last_peered":"2024-05-27T06:45:03.620222+0000","last_clean":"2024-05-27T06:45:03.620222+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:03.620222+0000","last_undegraded":"2024-05-27T06:45:03.620222+0000","last_fullsized":"2024-05-27T06:45:03.620222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541770+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:04.541770+0000","last_peered":"2024-05-27T06:45:04.541770+0000","last_clean":"2024-05-27T06:45:04.541770+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:04.541770+0000","last_undegraded":"2024-05-27T06:45:04.541770+0000","last_fullsized":"2024-05-27T06:45:04.541770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620170+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:03.620170+0000","last_peered":"2024-05-27T06:45:03.620170+0000","last_clean":"2024-05-27T06:45:03.620170+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:03.620170+0000","last_undegraded":"2024-05-27T06:45:03.620170+0000","last_fullsized":"2024-05-27T06:45:03.620170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542157+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:04.542157+0000","last_peered":"2024-05-27T06:45:04.542157+0000","last_clean":"2024-05-27T06:45:04.542157+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:04.542157+0000","last_undegraded":"2024-05-27T06:45:04.542157+0000","last_fullsized":"2024-05-27T06:45:04.542157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542660+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:04.542660+0000","last_peered":"2024-05-27T06:45:04.542660+0000","last_clean":"2024-05-27T06:45:04.542660+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:04.542660+0000","last_undegraded":"2024-05-27T06:45:04.542660+0000","last_fullsized":"2024-05-27T06:45:04.542660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619623+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:03.619623+0000","last_peered":"2024-05-27T06:45:03.619623+0000","last_clean":"2024-05-27T06:45:03.619623+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:03.619623+0000","last_undegraded":"2024-05-27T06:45:03.619623+0000","last_fullsized":"2024-05-27T06:45:03.619623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542082+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:04.542082+0000","last_peered":"2024-05-27T06:45:04.542082+0000","last_clean":"2024-05-27T06:45:04.542082+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:04.542082+0000","last_undegraded":"2024-05-27T06:45:04.542082+0000","last_fullsized":"2024-05-27T06:45:04.542082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619147+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:03.619147+0000","last_peered":"2024-05-27T06:45:03.619147+0000","last_clean":"2024-05-27T06:45:03.619147+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:03.619147+0000","last_undegraded":"2024-05-27T06:45:03.619147+0000","last_fullsized":"2024-05-27T06:45:03.619147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543112+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:04.543112+0000","last_peered":"2024-05-27T06:45:04.543112+0000","last_clean":"2024-05-27T06:45:04.543112+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:04.543112+0000","last_undegraded":"2024-05-27T06:45:04.543112+0000","last_fullsized":"2024-05-27T06:45:04.543112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542613+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:04.542613+0000","last_peered":"2024-05-27T06:45:04.542613+0000","last_clean":"2024-05-27T06:45:04.542613+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:04.542613+0000","last_undegraded":"2024-05-27T06:45:04.542613+0000","last_fullsized":"2024-05-27T06:45:04.542613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617693+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:03.617693+0000","last_peered":"2024-05-27T06:45:03.617693+0000","last_clean":"2024-05-27T06:45:03.617693+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:03.617693+0000","last_undegraded":"2024-05-27T06:45:03.617693+0000","last_fullsized":"2024-05-27T06:45:03.617693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541962+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:04.541962+0000","last_peered":"2024-05-27T06:45:04.541962+0000","last_clean":"2024-05-27T06:45:04.541962+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:04.541962+0000","last_undegraded":"2024-05-27T06:45:04.541962+0000","last_fullsized":"2024-05-27T06:45:04.541962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542858+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:04.542858+0000","last_peered":"2024-05-27T06:45:04.542858+0000","last_clean":"2024-05-27T06:45:04.542858+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:04.542858+0000","last_undegraded":"2024-05-27T06:45:04.542858+0000","last_fullsized":"2024-05-27T06:45:04.542858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619714+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:03.619714+0000","last_peered":"2024-05-27T06:45:03.619714+0000","last_clean":"2024-05-27T06:45:03.619714+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:03.619714+0000","last_undegraded":"2024-05-27T06:45:03.619714+0000","last_fullsized":"2024-05-27T06:45:03.619714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542030+0000","last_change":"2024-05-27T06:45:02.627394+0000","last_active":"2024-05-27T06:45:04.542030+0000","last_peered":"2024-05-27T06:45:04.542030+0000","last_clean":"2024-05-27T06:45:04.542030+0000","last_became_active":"2024-05-27T06:45:02.626550+0000","last_became_peered":"2024-05-27T06:45:02.626550+0000","last_unstale":"2024-05-27T06:45:04.542030+0000","last_undegraded":"2024-05-27T06:45:04.542030+0000","last_fullsized":"2024-05-27T06:45:04.542030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:01.383493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541845+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:04.541845+0000","last_peered":"2024-05-27T06:45:04.541845+0000","last_clean":"2024-05-27T06:45:04.541845+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:04.541845+0000","last_undegraded":"2024-05-27T06:45:04.541845+0000","last_fullsized":"2024-05-27T06:45:04.541845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542588+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:04.542588+0000","last_peered":"2024-05-27T06:45:04.542588+0000","last_clean":"2024-05-27T06:45:04.542588+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:04.542588+0000","last_undegraded":"2024-05-27T06:45:04.542588+0000","last_fullsized":"2024-05-27T06:45:04.542588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542786+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:04.542786+0000","last_peered":"2024-05-27T06:45:04.542786+0000","last_clean":"2024-05-27T06:45:04.542786+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:04.542786+0000","last_undegraded":"2024-05-27T06:45:04.542786+0000","last_fullsized":"2024-05-27T06:45:04.542786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543053+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:04.543053+0000","last_peered":"2024-05-27T06:45:04.543053+0000","last_clean":"2024-05-27T06:45:04.543053+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:04.543053+0000","last_undegraded":"2024-05-27T06:45:04.543053+0000","last_fullsized":"2024-05-27T06:45:04.543053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541946+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:04.541946+0000","last_peered":"2024-05-27T06:45:04.541946+0000","last_clean":"2024-05-27T06:45:04.541946+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:04.541946+0000","last_undegraded":"2024-05-27T06:45:04.541946+0000","last_fullsized":"2024-05-27T06:45:04.541946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541609+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:04.541609+0000","last_peered":"2024-05-27T06:45:04.541609+0000","last_clean":"2024-05-27T06:45:04.541609+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:04.541609+0000","last_undegraded":"2024-05-27T06:45:04.541609+0000","last_fullsized":"2024-05-27T06:45:04.541609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542424+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:04.542424+0000","last_peered":"2024-05-27T06:45:04.542424+0000","last_clean":"2024-05-27T06:45:04.542424+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:04.542424+0000","last_undegraded":"2024-05-27T06:45:04.542424+0000","last_fullsized":"2024-05-27T06:45:04.542424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619231+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:03.619231+0000","last_peered":"2024-05-27T06:45:03.619231+0000","last_clean":"2024-05-27T06:45:03.619231+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:03.619231+0000","last_undegraded":"2024-05-27T06:45:03.619231+0000","last_fullsized":"2024-05-27T06:45:03.619231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617259+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:03.617259+0000","last_peered":"2024-05-27T06:45:03.617259+0000","last_clean":"2024-05-27T06:45:03.617259+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:03.617259+0000","last_undegraded":"2024-05-27T06:45:03.617259+0000","last_fullsized":"2024-05-27T06:45:03.617259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542316+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:04.542316+0000","last_peered":"2024-05-27T06:45:04.542316+0000","last_clean":"2024-05-27T06:45:04.542316+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:04.542316+0000","last_undegraded":"2024-05-27T06:45:04.542316+0000","last_fullsized":"2024-05-27T06:45:04.542316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619378+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:03.619378+0000","last_peered":"2024-05-27T06:45:03.619378+0000","last_clean":"2024-05-27T06:45:03.619378+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:03.619378+0000","last_undegraded":"2024-05-27T06:45:03.619378+0000","last_fullsized":"2024-05-27T06:45:03.619378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619917+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:03.619917+0000","last_peered":"2024-05-27T06:45:03.619917+0000","last_clean":"2024-05-27T06:45:03.619917+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:03.619917+0000","last_undegraded":"2024-05-27T06:45:03.619917+0000","last_fullsized":"2024-05-27T06:45:03.619917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543182+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:04.543182+0000","last_peered":"2024-05-27T06:45:04.543182+0000","last_clean":"2024-05-27T06:45:04.543182+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:04.543182+0000","last_undegraded":"2024-05-27T06:45:04.543182+0000","last_fullsized":"2024-05-27T06:45:04.543182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641647+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:07.641647+0000","last_peered":"2024-05-27T06:45:07.641647+0000","last_clean":"2024-05-27T06:45:07.641647+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:07.641647+0000","last_undegraded":"2024-05-27T06:45:07.641647+0000","last_fullsized":"2024-05-27T06:45:07.641647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543150+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:04.543150+0000","last_peered":"2024-05-27T06:45:04.543150+0000","last_clean":"2024-05-27T06:45:04.543150+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:04.543150+0000","last_undegraded":"2024-05-27T06:45:04.543150+0000","last_fullsized":"2024-05-27T06:45:04.543150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.643949+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:07.643949+0000","last_peered":"2024-05-27T06:45:07.643949+0000","last_clean":"2024-05-27T06:45:07.643949+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:07.643949+0000","last_undegraded":"2024-05-27T06:45:07.643949+0000","last_fullsized":"2024-05-27T06:45:07.643949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618838+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:03.618838+0000","last_peered":"2024-05-27T06:45:03.618838+0000","last_clean":"2024-05-27T06:45:03.618838+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:03.618838+0000","last_undegraded":"2024-05-27T06:45:03.618838+0000","last_fullsized":"2024-05-27T06:45:03.618838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641947+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:07.641947+0000","last_peered":"2024-05-27T06:45:07.641947+0000","last_clean":"2024-05-27T06:45:07.641947+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:07.641947+0000","last_undegraded":"2024-05-27T06:45:07.641947+0000","last_fullsized":"2024-05-27T06:45:07.641947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542669+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:04.542669+0000","last_peered":"2024-05-27T06:45:04.542669+0000","last_clean":"2024-05-27T06:45:04.542669+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:04.542669+0000","last_undegraded":"2024-05-27T06:45:04.542669+0000","last_fullsized":"2024-05-27T06:45:04.542669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619690+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.619690+0000","last_peered":"2024-05-27T06:45:03.619690+0000","last_clean":"2024-05-27T06:45:03.619690+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:03.619690+0000","last_undegraded":"2024-05-27T06:45:03.619690+0000","last_fullsized":"2024-05-27T06:45:03.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619590+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:03.619590+0000","last_peered":"2024-05-27T06:45:03.619590+0000","last_clean":"2024-05-27T06:45:03.619590+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:03.619590+0000","last_undegraded":"2024-05-27T06:45:03.619590+0000","last_fullsized":"2024-05-27T06:45:03.619590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541708+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:04.541708+0000","last_peered":"2024-05-27T06:45:04.541708+0000","last_clean":"2024-05-27T06:45:04.541708+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:04.541708+0000","last_undegraded":"2024-05-27T06:45:04.541708+0000","last_fullsized":"2024-05-27T06:45:04.541708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541996+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:04.541996+0000","last_peered":"2024-05-27T06:45:04.541996+0000","last_clean":"2024-05-27T06:45:04.541996+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:04.541996+0000","last_undegraded":"2024-05-27T06:45:04.541996+0000","last_fullsized":"2024-05-27T06:45:04.541996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.643262+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:07.643262+0000","last_peered":"2024-05-27T06:45:07.643262+0000","last_clean":"2024-05-27T06:45:07.643262+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:07.643262+0000","last_undegraded":"2024-05-27T06:45:07.643262+0000","last_fullsized":"2024-05-27T06:45:07.643262+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619502+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:03.619502+0000","last_peered":"2024-05-27T06:45:03.619502+0000","last_clean":"2024-05-27T06:45:03.619502+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:03.619502+0000","last_undegraded":"2024-05-27T06:45:03.619502+0000","last_fullsized":"2024-05-27T06:45:03.619502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543172+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:04.543172+0000","last_peered":"2024-05-27T06:45:04.543172+0000","last_clean":"2024-05-27T06:45:04.543172+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:04.543172+0000","last_undegraded":"2024-05-27T06:45:04.543172+0000","last_fullsized":"2024-05-27T06:45:04.543172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620404+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:03.620404+0000","last_peered":"2024-05-27T06:45:03.620404+0000","last_clean":"2024-05-27T06:45:03.620404+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:03.620404+0000","last_undegraded":"2024-05-27T06:45:03.620404+0000","last_fullsized":"2024-05-27T06:45:03.620404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641718+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:07.641718+0000","last_peered":"2024-05-27T06:45:07.641718+0000","last_clean":"2024-05-27T06:45:07.641718+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:07.641718+0000","last_undegraded":"2024-05-27T06:45:07.641718+0000","last_fullsized":"2024-05-27T06:45:07.641718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620405+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:03.620405+0000","last_peered":"2024-05-27T06:45:03.620405+0000","last_clean":"2024-05-27T06:45:03.620405+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:03.620405+0000","last_undegraded":"2024-05-27T06:45:03.620405+0000","last_fullsized":"2024-05-27T06:45:03.620405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617687+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:03.617687+0000","last_peered":"2024-05-27T06:45:03.617687+0000","last_clean":"2024-05-27T06:45:03.617687+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:03.617687+0000","last_undegraded":"2024-05-27T06:45:03.617687+0000","last_fullsized":"2024-05-27T06:45:03.617687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619736+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:03.619736+0000","last_peered":"2024-05-27T06:45:03.619736+0000","last_clean":"2024-05-27T06:45:03.619736+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:03.619736+0000","last_undegraded":"2024-05-27T06:45:03.619736+0000","last_fullsized":"2024-05-27T06:45:03.619736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541528+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:04.541528+0000","last_peered":"2024-05-27T06:45:04.541528+0000","last_clean":"2024-05-27T06:45:04.541528+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:04.541528+0000","last_undegraded":"2024-05-27T06:45:04.541528+0000","last_fullsized":"2024-05-27T06:45:04.541528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620105+0000","last_change":"2024-05-27T06:45:02.651256+0000","last_active":"2024-05-27T06:45:03.620105+0000","last_peered":"2024-05-27T06:45:03.620105+0000","last_clean":"2024-05-27T06:45:03.620105+0000","last_became_active":"2024-05-27T06:45:02.650646+0000","last_became_peered":"2024-05-27T06:45:02.650646+0000","last_unstale":"2024-05-27T06:45:03.620105+0000","last_undegraded":"2024-05-27T06:45:03.620105+0000","last_fullsized":"2024-05-27T06:45:03.620105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:39:57.469688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618689+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:03.618689+0000","last_peered":"2024-05-27T06:45:03.618689+0000","last_clean":"2024-05-27T06:45:03.618689+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:03.618689+0000","last_undegraded":"2024-05-27T06:45:03.618689+0000","last_fullsized":"2024-05-27T06:45:03.618689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619124+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:03.619124+0000","last_peered":"2024-05-27T06:45:03.619124+0000","last_clean":"2024-05-27T06:45:03.619124+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:03.619124+0000","last_undegraded":"2024-05-27T06:45:03.619124+0000","last_fullsized":"2024-05-27T06:45:03.619124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541730+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:04.541730+0000","last_peered":"2024-05-27T06:45:04.541730+0000","last_clean":"2024-05-27T06:45:04.541730+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:04.541730+0000","last_undegraded":"2024-05-27T06:45:04.541730+0000","last_fullsized":"2024-05-27T06:45:04.541730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618565+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:03.618565+0000","last_peered":"2024-05-27T06:45:03.618565+0000","last_clean":"2024-05-27T06:45:03.618565+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:03.618565+0000","last_undegraded":"2024-05-27T06:45:03.618565+0000","last_fullsized":"2024-05-27T06:45:03.618565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543183+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:04.543183+0000","last_peered":"2024-05-27T06:45:04.543183+0000","last_clean":"2024-05-27T06:45:04.543183+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:04.543183+0000","last_undegraded":"2024-05-27T06:45:04.543183+0000","last_fullsized":"2024-05-27T06:45:04.543183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542094+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:04.542094+0000","last_peered":"2024-05-27T06:45:04.542094+0000","last_clean":"2024-05-27T06:45:04.542094+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:04.542094+0000","last_undegraded":"2024-05-27T06:45:04.542094+0000","last_fullsized":"2024-05-27T06:45:04.542094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617210+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:03.617210+0000","last_peered":"2024-05-27T06:45:03.617210+0000","last_clean":"2024-05-27T06:45:03.617210+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:03.617210+0000","last_undegraded":"2024-05-27T06:45:03.617210+0000","last_fullsized":"2024-05-27T06:45:03.617210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619757+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:03.619757+0000","last_peered":"2024-05-27T06:45:03.619757+0000","last_clean":"2024-05-27T06:45:03.619757+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:03.619757+0000","last_undegraded":"2024-05-27T06:45:03.619757+0000","last_fullsized":"2024-05-27T06:45:03.619757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542806+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:04.542806+0000","last_peered":"2024-05-27T06:45:04.542806+0000","last_clean":"2024-05-27T06:45:04.542806+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:04.542806+0000","last_undegraded":"2024-05-27T06:45:04.542806+0000","last_fullsized":"2024-05-27T06:45:04.542806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617419+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:03.617419+0000","last_peered":"2024-05-27T06:45:03.617419+0000","last_clean":"2024-05-27T06:45:03.617419+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:03.617419+0000","last_undegraded":"2024-05-27T06:45:03.617419+0000","last_fullsized":"2024-05-27T06:45:03.617419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619493+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:03.619493+0000","last_peered":"2024-05-27T06:45:03.619493+0000","last_clean":"2024-05-27T06:45:03.619493+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:03.619493+0000","last_undegraded":"2024-05-27T06:45:03.619493+0000","last_fullsized":"2024-05-27T06:45:03.619493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620147+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:03.620147+0000","last_peered":"2024-05-27T06:45:03.620147+0000","last_clean":"2024-05-27T06:45:03.620147+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:03.620147+0000","last_undegraded":"2024-05-27T06:45:03.620147+0000","last_fullsized":"2024-05-27T06:45:03.620147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617526+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:03.617526+0000","last_peered":"2024-05-27T06:45:03.617526+0000","last_clean":"2024-05-27T06:45:03.617526+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:03.617526+0000","last_undegraded":"2024-05-27T06:45:03.617526+0000","last_fullsized":"2024-05-27T06:45:03.617526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617705+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:03.617705+0000","last_peered":"2024-05-27T06:45:03.617705+0000","last_clean":"2024-05-27T06:45:03.617705+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:03.617705+0000","last_undegraded":"2024-05-27T06:45:03.617705+0000","last_fullsized":"2024-05-27T06:45:03.617705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543116+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:04.543116+0000","last_peered":"2024-05-27T06:45:04.543116+0000","last_clean":"2024-05-27T06:45:04.543116+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:04.543116+0000","last_undegraded":"2024-05-27T06:45:04.543116+0000","last_fullsized":"2024-05-27T06:45:04.543116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641734+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:07.641734+0000","last_peered":"2024-05-27T06:45:07.641734+0000","last_clean":"2024-05-27T06:45:07.641734+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:07.641734+0000","last_undegraded":"2024-05-27T06:45:07.641734+0000","last_fullsized":"2024-05-27T06:45:07.641734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542819+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:04.542819+0000","last_peered":"2024-05-27T06:45:04.542819+0000","last_clean":"2024-05-27T06:45:04.542819+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:04.542819+0000","last_undegraded":"2024-05-27T06:45:04.542819+0000","last_fullsized":"2024-05-27T06:45:04.542819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618413+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:03.618413+0000","last_peered":"2024-05-27T06:45:03.618413+0000","last_clean":"2024-05-27T06:45:03.618413+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:03.618413+0000","last_undegraded":"2024-05-27T06:45:03.618413+0000","last_fullsized":"2024-05-27T06:45:03.618413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542399+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:04.542399+0000","last_peered":"2024-05-27T06:45:04.542399+0000","last_clean":"2024-05-27T06:45:04.542399+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:04.542399+0000","last_undegraded":"2024-05-27T06:45:04.542399+0000","last_fullsized":"2024-05-27T06:45:04.542399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617281+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:03.617281+0000","last_peered":"2024-05-27T06:45:03.617281+0000","last_clean":"2024-05-27T06:45:03.617281+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:03.617281+0000","last_undegraded":"2024-05-27T06:45:03.617281+0000","last_fullsized":"2024-05-27T06:45:03.617281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542603+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:04.542603+0000","last_peered":"2024-05-27T06:45:04.542603+0000","last_clean":"2024-05-27T06:45:04.542603+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:04.542603+0000","last_undegraded":"2024-05-27T06:45:04.542603+0000","last_fullsized":"2024-05-27T06:45:04.542603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617374+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:03.617374+0000","last_peered":"2024-05-27T06:45:03.617374+0000","last_clean":"2024-05-27T06:45:03.617374+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:03.617374+0000","last_undegraded":"2024-05-27T06:45:03.617374+0000","last_fullsized":"2024-05-27T06:45:03.617374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543008+0000","last_change":"2024-05-27T06:45:02.632527+0000","last_active":"2024-05-27T06:45:04.543008+0000","last_peered":"2024-05-27T06:45:04.543008+0000","last_clean":"2024-05-27T06:45:04.543008+0000","last_became_active":"2024-05-27T06:45:02.632140+0000","last_became_peered":"2024-05-27T06:45:02.632140+0000","last_unstale":"2024-05-27T06:45:04.543008+0000","last_undegraded":"2024-05-27T06:45:04.543008+0000","last_fullsized":"2024-05-27T06:45:04.543008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.644734+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:07.644734+0000","last_peered":"2024-05-27T06:45:07.644734+0000","last_clean":"2024-05-27T06:45:07.644734+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:07.644734+0000","last_undegraded":"2024-05-27T06:45:07.644734+0000","last_fullsized":"2024-05-27T06:45:07.644734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541876+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:04.541876+0000","last_peered":"2024-05-27T06:45:04.541876+0000","last_clean":"2024-05-27T06:45:04.541876+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:04.541876+0000","last_undegraded":"2024-05-27T06:45:04.541876+0000","last_fullsized":"2024-05-27T06:45:04.541876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619831+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:03.619831+0000","last_peered":"2024-05-27T06:45:03.619831+0000","last_clean":"2024-05-27T06:45:03.619831+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:03.619831+0000","last_undegraded":"2024-05-27T06:45:03.619831+0000","last_fullsized":"2024-05-27T06:45:03.619831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.641892+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:07.641892+0000","last_peered":"2024-05-27T06:45:07.641892+0000","last_clean":"2024-05-27T06:45:07.641892+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:07.641892+0000","last_undegraded":"2024-05-27T06:45:07.641892+0000","last_fullsized":"2024-05-27T06:45:07.641892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619460+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:03.619460+0000","last_peered":"2024-05-27T06:45:03.619460+0000","last_clean":"2024-05-27T06:45:03.619460+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:03.619460+0000","last_undegraded":"2024-05-27T06:45:03.619460+0000","last_fullsized":"2024-05-27T06:45:03.619460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.642913+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:07.642913+0000","last_peered":"2024-05-27T06:45:07.642913+0000","last_clean":"2024-05-27T06:45:07.642913+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:07.642913+0000","last_undegraded":"2024-05-27T06:45:07.642913+0000","last_fullsized":"2024-05-27T06:45:07.642913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542992+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:04.542992+0000","last_peered":"2024-05-27T06:45:04.542992+0000","last_clean":"2024-05-27T06:45:04.542992+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:04.542992+0000","last_undegraded":"2024-05-27T06:45:04.542992+0000","last_fullsized":"2024-05-27T06:45:04.542992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542987+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:04.542987+0000","last_peered":"2024-05-27T06:45:04.542987+0000","last_clean":"2024-05-27T06:45:04.542987+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:04.542987+0000","last_undegraded":"2024-05-27T06:45:04.542987+0000","last_fullsized":"2024-05-27T06:45:04.542987+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541662+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:04.541662+0000","last_peered":"2024-05-27T06:45:04.541662+0000","last_clean":"2024-05-27T06:45:04.541662+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:04.541662+0000","last_undegraded":"2024-05-27T06:45:04.541662+0000","last_fullsized":"2024-05-27T06:45:04.541662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.644507+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:07.644507+0000","last_peered":"2024-05-27T06:45:07.644507+0000","last_clean":"2024-05-27T06:45:07.644507+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:07.644507+0000","last_undegraded":"2024-05-27T06:45:07.644507+0000","last_fullsized":"2024-05-27T06:45:07.644507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":666555,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":294912,"data_stored":666611,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:13.934 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-27T06:45:13.934 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-27T06:45:13.934 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-27T06:45:13.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 Processor -- start 2024-05-27T06:45:14.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 -- start start 2024-05-27T06:45:14.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4c54154200 0x7f4c54154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c540c78b0 con 0x7f4c540cbd00 2024-05-27T06:45:14.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4c54154b50 con 0x7f4c54154200 2024-05-27T06:45:14.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c59000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4c540c6ee0 con 0x7f4c540ca770 2024-05-27T06:45:14.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c52800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4c54154200 0x7f4c54154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c51400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4c540cbd00 0x7f4c540cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59238/0 (socket says 172.21.15.81:59238) 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55166/0 (socket says 172.21.15.81:55166) 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c51400640 1 -- 172.21.15.81:0/1990485187 learned_addr learned my addr 172.21.15.81:0/1990485187 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c52800640 1 -- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 msgr2=0x7f4c540cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c52800640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c52800640 1 -- 172.21.15.81:0/1990485187 >> v1:172.21.15.106:6789/0 conn(0x7f4c540cbd00 legacy=0x7f4c540cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c52800640 1 -- 172.21.15.81:0/1990485187 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c540c71c0 con 0x7f4c54154200 2024-05-27T06:45:14.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.126+0000 7f4c51e00640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:14.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c52800640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c54154200 0x7f4c54154610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097c0 tx=0x7f4c4c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b0f8fd28947b64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:14.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c50a00640 1 -- 172.21.15.81:0/1990485187 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0b1070 con 0x7f4c54154200 2024-05-27T06:45:14.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c50a00640 1 -- 172.21.15.81:0/1990485187 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c4c0ac410 con 0x7f4c54154200 2024-05-27T06:45:14.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c50a00640 1 -- 172.21.15.81:0/1990485187 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0ac7e0 con 0x7f4c54154200 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 -- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c54154200 msgr2=0x7f4c54154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c54154200 0x7f4c54154610 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0097c0 tx=0x7f4c4c01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 -- 172.21.15.81:0/1990485187 shutdown_connections 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c54154200 0x7f4c54154610 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 --2- 172.21.15.81:0/1990485187 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c540ca770 0x7f4c540cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 -- 172.21.15.81:0/1990485187 >> 172.21.15.81:0/1990485187 conn(0x7f4c54068ae0 msgr2=0x7f4c54068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 -- 172.21.15.81:0/1990485187 shutdown_connections 2024-05-27T06:45:14.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.130+0000 7f4c59000640 1 -- 172.21.15.81:0/1990485187 wait complete. 2024-05-27T06:45:14.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 Processor -- start 2024-05-27T06:45:14.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 -- start start 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 0x7f4c54150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 0x7f4c541f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540d0640 con 0x7f4c540ca770 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4c540d04c0 con 0x7f4c540cbd00 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c59000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4c54167590 con 0x7f4c54154200 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57628/0 (socket says 172.21.15.81:57628) 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.134+0000 7f4c51e00640 1 -- 172.21.15.81:0/1532221287 learned_addr learned my addr 172.21.15.81:0/1532221287 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c52800640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 0x7f4c541f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51400640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 0x7f4c54150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 -- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 msgr2=0x7f4c541f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 0x7f4c541f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 -- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 msgr2=0x7f4c54150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 0x7f4c54150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 -- 172.21.15.81:0/1532221287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c541f3c70 con 0x7f4c540ca770 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c52800640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 0x7f4c541f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51400640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 0x7f4c54150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:14.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c51e00640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400efc0 tx=0x7f4c4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44018020 con 0x7f4c540ca770 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c44014680 con 0x7f4c540ca770 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c541f4670 con 0x7f4c540ca770 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c44014a50 con 0x7f4c540ca770 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c541600e0 con 0x7f4c540ca770 2024-05-27T06:45:14.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5406a560 con 0x7f4c540ca770 2024-05-27T06:45:14.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4c4401b020 con 0x7f4c540ca770 2024-05-27T06:45:14.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.138+0000 7f4c4a000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 0x7f4c34071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:14.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.142+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4c44090040 con 0x7f4c540ca770 2024-05-27T06:45:14.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.142+0000 7f4c51400640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 0x7f4c34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:14.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.142+0000 7f4c51400640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 0x7f4c34071910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4c540c6b70 tx=0x7f4c40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:14.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.146+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4c44022030 con 0x7f4c540ca770 2024-05-27T06:45:14.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.334+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4c540727d0 con 0x7f4c540ca770 2024-05-27T06:45:14.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.334+0000 7f4c4a000640 1 -- 172.21.15.81:0/1532221287 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f4c541f3c70 con 0x7f4c540ca770 2024-05-27T06:45:14.329 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:14.329 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-27T06:45:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 msgr2=0x7f4c34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 0x7f4c34071910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4c540c6b70 tx=0x7f4c40009040 comp rx=0 tx=0).stop 2024-05-27T06:45:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 msgr2=0x7f4c541f4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400efc0 tx=0x7f4c4400c490 comp rx=0 tx=0).stop 2024-05-27T06:45:14.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 shutdown_connections 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4c54154200 0x7f4c541f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4c3406f400 0x7f4c34071910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c540cbd00 0x7f4c54150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 --2- 172.21.15.81:0/1532221287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540ca770 0x7f4c541f4130 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 >> 172.21.15.81:0/1532221287 conn(0x7f4c54068ae0 msgr2=0x7f4c54068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 shutdown_connections 2024-05-27T06:45:14.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:14.338+0000 7f4c59000640 1 -- 172.21.15.81:0/1532221287 wait complete. 2024-05-27T06:45:17.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:17.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.518+0000 7f5cf6200640 1 Processor -- start 2024-05-27T06:45:17.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.518+0000 7f5cf6200640 1 -- start start 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cf6200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5cf0072430 0x7f5cf0159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cf6200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf0072840 con 0x7f5cf0161cd0 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cf6200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf00c5270 con 0x7f5cf015a040 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cf6200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf00c5040 con 0x7f5cf0072430 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cefe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5cf0161cd0 0x7f5cf01640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46644/0 (socket says 172.21.15.81:46644) 2024-05-27T06:45:17.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cefe00640 1 -- 172.21.15.81:0/1988456790 learned_addr learned my addr 172.21.15.81:0/1988456790 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cef400640 1 --2- 172.21.15.81:0/1988456790 >> v2:172.21.15.106:3300/0 conn(0x7f5cf0072430 0x7f5cf0159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3968287035 0 0) 0x7f5cf0072840 con 0x7f5cf0161cd0 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd4005740 con 0x7f5cf0161cd0 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 994726047 0 0) 0x7f5cd4005740 con 0x7f5cf0161cd0 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cef400640 1 -- 172.21.15.81:0/1988456790 >> v1:172.21.15.106:6789/0 conn(0x7f5cf015a040 legacy=0x7f5cf00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cef400640 1 -- 172.21.15.81:0/1988456790 >> v1:172.21.15.81:6789/0 conn(0x7f5cf0161cd0 legacy=0x7f5cf01640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.522+0000 7f5cef400640 1 -- 172.21.15.81:0/1988456790 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00c7810 con 0x7f5cf0072430 2024-05-27T06:45:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cef400640 1 --2- 172.21.15.81:0/1988456790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf0072430 0x7f5cf0159b00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00097c0 tx=0x7f5ce001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fe5095b6c00f681 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce00b0070 con 0x7f5cf0072430 2024-05-27T06:45:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce00ab400 con 0x7f5cf0072430 2024-05-27T06:45:17.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5ceea00640 1 -- 172.21.15.81:0/1988456790 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8001a80 con 0x7f5cf0072430 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 -- 172.21.15.81:0/1988456790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf0072430 msgr2=0x7f5cf0159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 --2- 172.21.15.81:0/1988456790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf0072430 0x7f5cf0159b00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00097c0 tx=0x7f5ce001c3c0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 -- 172.21.15.81:0/1988456790 shutdown_connections 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 --2- 172.21.15.81:0/1988456790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf0072430 0x7f5cf0159b00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 -- 172.21.15.81:0/1988456790 >> 172.21.15.81:0/1988456790 conn(0x7f5cf0070700 msgr2=0x7f5cf00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 -- 172.21.15.81:0/1988456790 shutdown_connections 2024-05-27T06:45:17.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.526+0000 7f5cf6200640 1 -- 172.21.15.81:0/1988456790 wait complete. 2024-05-27T06:45:17.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 Processor -- start 2024-05-27T06:45:17.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- start start 2024-05-27T06:45:17.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cf0072430 0x7f5cf01f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:17.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf015a040 0x7f5cf01622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:17.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00c7dd0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00c75c0 con 0x7f5cf015a040 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00c5610 con 0x7f5cf0072430 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41898/0 (socket says 172.21.15.81:41898) 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 -- 172.21.15.81:0/4081028569 learned_addr learned my addr 172.21.15.81:0/4081028569 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cef400640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cf0072430 0x7f5cf01f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5ce6a00640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf015a040 0x7f5cf01622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 -- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cf0072430 msgr2=0x7f5cf01f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cf0072430 0x7f5cf01f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 -- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf015a040 msgr2=0x7f5cf01622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf015a040 0x7f5cf01622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 -- 172.21.15.81:0/4081028569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf01f6870 con 0x7f5cf0161cd0 2024-05-27T06:45:17.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cefe00640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800cb70 tx=0x7f5ce800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:17.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800ace0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce8012400 con 0x7f5cf0161cd0 2024-05-27T06:45:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf01f9820 con 0x7f5cf0161cd0 2024-05-27T06:45:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80127f0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.530+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf01f9de0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf006a560 con 0x7f5cf0161cd0 2024-05-27T06:45:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ce8012ba0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5ce7e00640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 0x7f5ccc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:17.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ce808f9e0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5cef400640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 0x7f5ccc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:17.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.534+0000 7f5cef400640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 0x7f5ccc071910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00ad000 tx=0x7f5ce00a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:17.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.538+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5ce805c300 con 0x7f5cf0161cd0 2024-05-27T06:45:17.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5cf00c4ae0 con 0x7f5cf0161cd0 2024-05-27T06:45:17.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5ce7e00640 1 -- 172.21.15.81:0/4081028569 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f5ce8060600 con 0x7f5cf0161cd0 2024-05-27T06:45:17.732 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:17.732 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 msgr2=0x7f5ccc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 0x7f5ccc071910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5ce00ad000 tx=0x7f5ce00a6000 comp rx=0 tx=0).stop 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 msgr2=0x7f5cf015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800cb70 tx=0x7f5ce800a430 comp rx=0 tx=0).stop 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 shutdown_connections 2024-05-27T06:45:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ccc06f400 0x7f5ccc071910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf0161cd0 0x7f5cf015cea0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cf015a040 0x7f5cf01622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 --2- 172.21.15.81:0/4081028569 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cf0072430 0x7f5cf01f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 >> 172.21.15.81:0/4081028569 conn(0x7f5cf0070700 msgr2=0x7f5cf00c67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 shutdown_connections 2024-05-27T06:45:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:17.738+0000 7f5cf6200640 1 -- 172.21.15.81:0/4081028569 wait complete. 2024-05-27T06:45:17.993 INFO:tasks.ceph.mds.f.smithi106.stderr:2024-05-27T06:45:17.995+0000 7ff48a600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-27T06:45:17.993 INFO:tasks.ceph.mds.f.smithi106.stderr:2024-05-27T06:45:17.995+0000 7ff48a600640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-05-27T06:45:20.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:20.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.918+0000 7f5cb5e00640 1 Processor -- start 2024-05-27T06:45:20.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.918+0000 7f5cb5e00640 1 -- start start 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.918+0000 7f5cb5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5cb00c5b60 0x7f5cb00c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cb5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb0161cd0 0x7f5cb01640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cb5e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb0072840 con 0x7f5cb0072430 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cb5e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f5cb00c5040 con 0x7f5cb00c5b60 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cb5e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb00c75c0 con 0x7f5cb0161cd0 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5caea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5cb0072430 0x7f5cb0159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46646/0 (socket says 172.21.15.81:46646) 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5caea00640 1 -- 172.21.15.81:0/1700947871 learned_addr learned my addr 172.21.15.81:0/1700947871 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 --2- 172.21.15.81:0/1700947871 >> v2:172.21.15.81:3300/0 conn(0x7f5cb00c5b60 0x7f5cb00c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5caf400640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb0161cd0 0x7f5cb01640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cae000640 1 -- 172.21.15.81:0/1700947871 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963081363 0 0) 0x7f5cb0072840 con 0x7f5cb0072430 2024-05-27T06:45:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cae000640 1 -- 172.21.15.81:0/1700947871 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c94003530 con 0x7f5cb0072430 2024-05-27T06:45:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 -- 172.21.15.81:0/1700947871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb0161cd0 msgr2=0x7f5cb01640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb0161cd0 0x7f5cb01640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 -- 172.21.15.81:0/1700947871 >> v1:172.21.15.81:6789/0 conn(0x7f5cb0072430 legacy=0x7f5cb0159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 -- 172.21.15.81:0/1700947871 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb00c5270 con 0x7f5cb00c5b60 2024-05-27T06:45:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5ca6000640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb00c5b60 0x7f5cb00c5f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5c980097c0 tx=0x7f5c9801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c530fbae10f6250e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cae000640 1 -- 172.21.15.81:0/1700947871 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c980b0070 con 0x7f5cb00c5b60 2024-05-27T06:45:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cae000640 1 -- 172.21.15.81:0/1700947871 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c980ab600 con 0x7f5cb00c5b60 2024-05-27T06:45:20.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.922+0000 7f5cae000640 1 -- 172.21.15.81:0/1700947871 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c980ab9d0 con 0x7f5cb00c5b60 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1700947871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb00c5b60 msgr2=0x7f5cb00c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb00c5b60 0x7f5cb00c5f70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5c980097c0 tx=0x7f5c9801c4a0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1700947871 shutdown_connections 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb0161cd0 0x7f5cb01640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1700947871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb00c5b60 0x7f5cb00c5f70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1700947871 >> 172.21.15.81:0/1700947871 conn(0x7f5cb0070700 msgr2=0x7f5cb00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1700947871 shutdown_connections 2024-05-27T06:45:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1700947871 wait complete. 2024-05-27T06:45:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 Processor -- start 2024-05-27T06:45:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.926+0000 7f5cb5e00640 1 -- start start 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cb0072430 0x7f5cb0151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb00c5b60 0x7f5cb01f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb00ce6f0 con 0x7f5cb0161cd0 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb00c7dd0 con 0x7f5cb0072430 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb00c5610 con 0x7f5cb00c5b60 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41916/0 (socket says 172.21.15.81:41916) 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 -- 172.21.15.81:0/1272813930 learned_addr learned my addr 172.21.15.81:0/1272813930 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5ca6000640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb00c5b60 0x7f5cb01f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caea00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cb0072430 0x7f5cb0151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 -- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb00c5b60 msgr2=0x7f5cb01f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb00c5b60 0x7f5cb01f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 -- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cb0072430 msgr2=0x7f5cb0151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cb0072430 0x7f5cb0151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 -- 172.21.15.81:0/1272813930 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb01f5800 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5caf400640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8007d80 tx=0x7f5ca800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8014420 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ca8019070 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb0163630 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8014800 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb0163bb0 con 0x7f5cb0161cd0 2024-05-27T06:45:20.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.930+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb006a560 con 0x7f5cb0161cd0 2024-05-27T06:45:20.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.934+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ca80149a0 con 0x7f5cb0161cd0 2024-05-27T06:45:20.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.934+0000 7f5ca7e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 0x7f5c8c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:20.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.934+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5ca808fa30 con 0x7f5cb0161cd0 2024-05-27T06:45:20.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.934+0000 7f5caea00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 0x7f5c8c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:20.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.934+0000 7f5caea00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 0x7f5c8c071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0007a70 tx=0x7f5ca00079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:20.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:20.938+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5ca805c230 con 0x7f5cb0161cd0 2024-05-27T06:45:21.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5cb0152e80 con 0x7f5cb0161cd0 2024-05-27T06:45:21.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5ca7e00640 1 -- 172.21.15.81:0/1272813930 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f5ca8060530 con 0x7f5cb0161cd0 2024-05-27T06:45:21.124 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:21.124 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-27T06:45:21.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 msgr2=0x7f5c8c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:21.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 0x7f5c8c071910 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0007a70 tx=0x7f5ca00079c0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 msgr2=0x7f5cb013e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8007d80 tx=0x7f5ca800e4b0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 shutdown_connections 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5cb00c5b60 0x7f5cb01f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5c8c06f400 0x7f5c8c071910 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cb0161cd0 0x7f5cb013e590 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 --2- 172.21.15.81:0/1272813930 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5cb0072430 0x7f5cb0151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 >> 172.21.15.81:0/1272813930 conn(0x7f5cb0070700 msgr2=0x7f5cb014b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 shutdown_connections 2024-05-27T06:45:21.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:21.130+0000 7f5cb5e00640 1 -- 172.21.15.81:0/1272813930 wait complete. 2024-05-27T06:45:24.146 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.310+0000 7f6066e00640 1 Processor -- start 2024-05-27T06:45:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 -- start start 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f6060152f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f60600c46c0 0x7f60600c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60600c9aa0 con 0x7f60600c3e00 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f60600c9850 con 0x7f60600c46c0 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f60600c3c30 con 0x7f6060074a80 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6065000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f60600c3e00 0x7f60600ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46662/0 (socket says 172.21.15.81:46662) 2024-05-27T06:45:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6065000640 1 -- 172.21.15.81:0/3118369898 learned_addr learned my addr 172.21.15.81:0/3118369898 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066400640 1 --2- 172.21.15.81:0/3118369898 >> v2:172.21.15.81:3300/0 conn(0x7f60600c46c0 0x7f60600c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6065a00640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f6060152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066400640 1 -- 172.21.15.81:0/3118369898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 msgr2=0x7f6060152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066400640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f6060152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066400640 1 -- 172.21.15.81:0/3118369898 >> v1:172.21.15.81:6789/0 conn(0x7f60600c3e00 legacy=0x7f60600ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.314+0000 7f6066400640 1 -- 172.21.15.81:0/3118369898 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60600c9d80 con 0x7f60600c46c0 2024-05-27T06:45:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066400640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c46c0 0x7f60600c4ad0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6058009d90 tx=0x7f605801f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01cf6ad7a9d7c9d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f605fe00640 1 -- 172.21.15.81:0/3118369898 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60580b4070 con 0x7f60600c46c0 2024-05-27T06:45:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f605fe00640 1 -- 172.21.15.81:0/3118369898 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60580af600 con 0x7f60600c46c0 2024-05-27T06:45:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f605fe00640 1 -- 172.21.15.81:0/3118369898 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60580af9d0 con 0x7f60600c46c0 2024-05-27T06:45:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 -- 172.21.15.81:0/3118369898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c46c0 msgr2=0x7f60600c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c46c0 0x7f60600c4ad0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6058009d90 tx=0x7f605801f170 comp rx=0 tx=0).stop 2024-05-27T06:45:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 -- 172.21.15.81:0/3118369898 shutdown_connections 2024-05-27T06:45:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c46c0 0x7f60600c4ad0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 --2- 172.21.15.81:0/3118369898 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f6060152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 -- 172.21.15.81:0/3118369898 >> 172.21.15.81:0/3118369898 conn(0x7f606006c990 msgr2=0x7f606006cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 -- 172.21.15.81:0/3118369898 shutdown_connections 2024-05-27T06:45:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.318+0000 7f6066e00640 1 -- 172.21.15.81:0/3118369898 wait complete. 2024-05-27T06:45:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 Processor -- start 2024-05-27T06:45:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- start start 2024-05-27T06:45:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f606014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60600c46c0 0x7f6060157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60600c5010 con 0x7f60600c3e00 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f60600c3950 con 0x7f60600c46c0 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f60600ce9a0 con 0x7f6060074a80 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41922/0 (socket says 172.21.15.81:41922) 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 -- 172.21.15.81:0/884543684 learned_addr learned my addr 172.21.15.81:0/884543684 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066400640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60600c46c0 0x7f6060157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065a00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f606014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 -- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 msgr2=0x7f606014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f606014b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 -- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60600c46c0 msgr2=0x7f6060157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60600c46c0 0x7f6060157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 -- 172.21.15.81:0/884543684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606014d520 con 0x7f60600c3e00 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065a00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f606014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6065000640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f604c00c3c0 tx=0x7f604c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c017070 con 0x7f60600c3e00 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604c005d10 con 0x7f60600c3e00 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606014d280 con 0x7f60600c3e00 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c007310 con 0x7f60600c3e00 2024-05-27T06:45:24.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.322+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60601e3010 con 0x7f60600c3e00 2024-05-27T06:45:24.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6060050870 con 0x7f60600c3e00 2024-05-27T06:45:24.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f604c01a070 con 0x7f60600c3e00 2024-05-27T06:45:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f605d600640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f604c08e550 con 0x7f60600c3e00 2024-05-27T06:45:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f6065a00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:24.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.326+0000 7f6065a00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 0x7f6040071910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f605400a970 tx=0x7f6054008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:24.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.330+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f604c05b430 con 0x7f60600c3e00 2024-05-27T06:45:24.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f606014f1d0 con 0x7f60600c3e00 2024-05-27T06:45:24.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f605d600640 1 -- 172.21.15.81:0/884543684 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f604c05ad30 con 0x7f60600c3e00 2024-05-27T06:45:24.518 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:24.518 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-27T06:45:24.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 msgr2=0x7f6040071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 0x7f6040071910 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f605400a970 tx=0x7f6054008040 comp rx=0 tx=0).stop 2024-05-27T06:45:24.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 msgr2=0x7f6060157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:24.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.522+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f604c00c3c0 tx=0x7f604c00c880 comp rx=0 tx=0).stop 2024-05-27T06:45:24.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 shutdown_connections 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f604006f400 0x7f6040071910 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f60600c46c0 0x7f6060157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60600c3e00 0x7f6060157950 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 --2- 172.21.15.81:0/884543684 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6060074a80 0x7f606014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 >> 172.21.15.81:0/884543684 conn(0x7f606006c990 msgr2=0x7f6060158b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 shutdown_connections 2024-05-27T06:45:24.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:24.526+0000 7f6066e00640 1 -- 172.21.15.81:0/884543684 wait complete. 2024-05-27T06:45:27.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 Processor -- start 2024-05-27T06:45:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 -- start start 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef600c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef60072840 con 0x7fef60161cd0 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef600c5270 con 0x7fef60072430 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef65200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fef600c5040 con 0x7fef600c5b60 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5e800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fef60161cd0 0x7fef601640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34574/0 (socket says 172.21.15.81:34574) 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5e800640 1 -- 172.21.15.81:0/3058613011 learned_addr learned my addr 172.21.15.81:0/3058613011 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:27.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef57e00640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef600c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740190966 0 0) 0x7fef60072840 con 0x7fef60161cd0 2024-05-27T06:45:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef44003530 con 0x7fef60161cd0 2024-05-27T06:45:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562815140 0 0) 0x7fef600c5270 con 0x7fef60072430 2024-05-27T06:45:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef60072840 con 0x7fef60072430 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 566560922 0 0) 0x7fef44003530 con 0x7fef60161cd0 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 msgr2=0x7fef600c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef600c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 >> v1:172.21.15.106:6789/0 conn(0x7fef60072430 legacy=0x7fef60159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600c7810 con 0x7fef60161cd0 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.705+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fef58001a80 con 0x7fef60161cd0 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 >> v1:172.21.15.81:6789/0 conn(0x7fef60161cd0 legacy=0x7fef601640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef44003c20 0x7fef44004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef44004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef44005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef4400ddc0 con 0x7fef44004da0 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fef4400df70 con 0x7fef44003c20 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fef4400e170 con 0x7fef600c5b60 2024-05-27T06:45:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef44005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5de00640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef44003c20 0x7fef44004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef57e00640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef44004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 -- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 msgr2=0x7fef44004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef600c5b60 0x7fef44004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 -- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef44003c20 msgr2=0x7fef44004070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef44003c20 0x7fef44004070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 -- 172.21.15.81:0/3058613011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef44003530 con 0x7fef44004da0 2024-05-27T06:45:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5e800640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef44005230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a7c0 tx=0x7fef5801f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d7d02837e956c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5800a410 con 0x7fef44004da0 2024-05-27T06:45:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef580aa670 con 0x7fef44004da0 2024-05-27T06:45:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.709+0000 7fef5d400640 1 -- 172.21.15.81:0/3058613011 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef580aaa60 con 0x7fef44004da0 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- 172.21.15.81:0/3058613011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 msgr2=0x7fef44005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 --2- 172.21.15.81:0/3058613011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef44005230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fef5800a7c0 tx=0x7fef5801f9a0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef5e800640 1 -- 172.21.15.81:0/3058613011 reap_dead start 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- 172.21.15.81:0/3058613011 shutdown_connections 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- 172.21.15.81:0/3058613011 >> 172.21.15.81:0/3058613011 conn(0x7fef60070700 msgr2=0x7fef600cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- 172.21.15.81:0/3058613011 shutdown_connections 2024-05-27T06:45:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- 172.21.15.81:0/3058613011 wait complete. 2024-05-27T06:45:27.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 Processor -- start 2024-05-27T06:45:27.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.713+0000 7fef65200640 1 -- start start 2024-05-27T06:45:27.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef60151340 0x7fef601f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 0x7fef6014a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef600c7dd0 con 0x7fef44004da0 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fef600c75c0 con 0x7fef601f1de0 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fef601595e0 con 0x7fef60151340 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35108/0 (socket says 172.21.15.81:35108) 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 -- 172.21.15.81:0/242544912 learned_addr learned my addr 172.21.15.81:0/242544912 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef57e00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef60151340 0x7fef601f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5e800640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 0x7fef6014a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 -- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef60151340 msgr2=0x7fef601f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef60151340 0x7fef601f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 -- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 msgr2=0x7fef6014a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 0x7fef6014a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 -- 172.21.15.81:0/242544912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef600c6d30 con 0x7fef44004da0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5e800640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 0x7fef6014a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef5de00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef5000f820 tx=0x7fef5000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef500050d0 con 0x7fef44004da0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef50018070 con 0x7fef44004da0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef601ee330 con 0x7fef44004da0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5000d6d0 con 0x7fef44004da0 2024-05-27T06:45:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef601ee5f0 con 0x7fef44004da0 2024-05-27T06:45:27.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef6006a560 con 0x7fef44004da0 2024-05-27T06:45:27.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.717+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fef5000d870 con 0x7fef44004da0 2024-05-27T06:45:27.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.721+0000 7fef56000640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:27.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.721+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef5008f2a0 con 0x7fef44004da0 2024-05-27T06:45:27.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.721+0000 7fef57e00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:27.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.721+0000 7fef57e00640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 0x7fef3c071910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef601508c0 tx=0x7fef48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:27.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.725+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fef600c6d30 con 0x7fef44004da0 2024-05-27T06:45:27.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fef600c5940 con 0x7fef44004da0 2024-05-27T06:45:27.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef56000640 1 -- 172.21.15.81:0/242544912 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fef5005c0f0 con 0x7fef44004da0 2024-05-27T06:45:27.912 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:27.913 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 msgr2=0x7fef3c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 0x7fef3c071910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef601508c0 tx=0x7fef48008040 comp rx=0 tx=0).stop 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 msgr2=0x7fef601485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef5000f820 tx=0x7fef5000fce0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 shutdown_connections 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fef60151340 0x7fef601f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fef3c06f400 0x7fef3c071910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef601f1de0 0x7fef6014a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 --2- 172.21.15.81:0/242544912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef44004da0 0x7fef601485a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:27.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 >> 172.21.15.81:0/242544912 conn(0x7fef60070700 msgr2=0x7fef600c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:27.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 shutdown_connections 2024-05-27T06:45:27.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:27.917+0000 7fef65200640 1 -- 172.21.15.81:0/242544912 wait complete. 2024-05-27T06:45:28.198 INFO:tasks.ceph.mds.f.smithi106.stderr:2024-05-27T06:45:28.203+0000 7ff48d800640 -1 quiesce.mds.0 failed to decode message of type 1285 v1: End of buffer [buffer:2] 2024-05-27T06:45:30.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-27T06:45:31.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 Processor -- start 2024-05-27T06:45:31.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 -- start start 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda840c5f70 0x7fda84165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda84072840 con 0x7fda84072430 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840c5270 con 0x7fda840c5b60 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.097+0000 7fda8ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fda840c5040 con 0x7fda840c5f70 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda83e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fda84072430 0x7fda84159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34584/0 (socket says 172.21.15.81:34584) 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda83e00640 1 -- 172.21.15.81:0/3602050 learned_addr learned my addr 172.21.15.81:0/3602050 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:31.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda88e00640 1 --2- 172.21.15.81:0/3602050 >> v2:172.21.15.81:3300/0 conn(0x7fda840c5f70 0x7fda84165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda82a00640 1 -- 172.21.15.81:0/3602050 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978476945 0 0) 0x7fda84072840 con 0x7fda84072430 2024-05-27T06:45:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda82a00640 1 -- 172.21.15.81:0/3602050 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda68003530 con 0x7fda84072430 2024-05-27T06:45:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda88e00640 1 -- 172.21.15.81:0/3602050 >> v1:172.21.15.106:6789/0 conn(0x7fda840c5b60 legacy=0x7fda84161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda88e00640 1 -- 172.21.15.81:0/3602050 >> v1:172.21.15.81:6789/0 conn(0x7fda84072430 legacy=0x7fda84159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda88e00640 1 -- 172.21.15.81:0/3602050 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840c7810 con 0x7fda840c5f70 2024-05-27T06:45:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda88e00640 1 --2- 172.21.15.81:0/3602050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda84165400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009d90 tx=0x7fda7c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c7fc502beff00b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda82a00640 1 -- 172.21.15.81:0/3602050 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0b2070 con 0x7fda840c5f70 2024-05-27T06:45:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda82a00640 1 -- 172.21.15.81:0/3602050 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c0a5830 con 0x7fda840c5f70 2024-05-27T06:45:31.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.101+0000 7fda82a00640 1 -- 172.21.15.81:0/3602050 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0a5c00 con 0x7fda840c5f70 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- 172.21.15.81:0/3602050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 msgr2=0x7fda84165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 --2- 172.21.15.81:0/3602050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda84165400 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fda7c009d90 tx=0x7fda7c01c970 comp rx=0 tx=0).stop 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- 172.21.15.81:0/3602050 shutdown_connections 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 --2- 172.21.15.81:0/3602050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda84165400 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- 172.21.15.81:0/3602050 >> 172.21.15.81:0/3602050 conn(0x7fda84070700 msgr2=0x7fda840cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- 172.21.15.81:0/3602050 shutdown_connections 2024-05-27T06:45:31.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- 172.21.15.81:0/3602050 wait complete. 2024-05-27T06:45:31.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 Processor -- start 2024-05-27T06:45:31.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.105+0000 7fda8ae00640 1 -- start start 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 0x7fda841f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 0x7fda8414dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fda840c7dd0 con 0x7fda840c5f70 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fda840c75c0 con 0x7fda84072430 2024-05-27T06:45:31.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fda840ce6f0 con 0x7fda840c5b60 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35120/0 (socket says 172.21.15.81:35120) 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 -- 172.21.15.81:0/2851736812 learned_addr learned my addr 172.21.15.81:0/2851736812 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda83e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 0x7fda841f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda83400640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 0x7fda8414dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 -- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 msgr2=0x7fda8414dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 0x7fda8414dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 -- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 msgr2=0x7fda841f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 0x7fda841f9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 -- 172.21.15.81:0/2851736812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda841fa480 con 0x7fda840c5f70 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda83e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 0x7fda841f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:31.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda83400640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 0x7fda8414dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:31.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda88e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fda7c01a5a0 tx=0x7fda7c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0b2040 con 0x7fda840c5f70 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c0bf070 con 0x7fda840c5f70 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda84149cc0 con 0x7fda840c5f70 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0b8670 con 0x7fda840c5f70 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda84149f60 con 0x7fda840c5f70 2024-05-27T06:45:31.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.109+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8406a560 con 0x7fda840c5f70 2024-05-27T06:45:31.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.113+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fda7c0b8810 con 0x7fda840c5f70 2024-05-27T06:45:31.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.113+0000 7fda7be00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 0x7fda64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.113+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fda7c129ca0 con 0x7fda840c5f70 2024-05-27T06:45:31.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.113+0000 7fda83e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 0x7fda64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.113+0000 7fda83e00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 0x7fda64071940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda74005890 tx=0x7fda74009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.117+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fda7c0c8020 con 0x7fda840c5f70 2024-05-27T06:45:31.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.305+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fda840c5940 con 0x7fda840c5f70 2024-05-27T06:45:31.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda7be00640 1 -- 172.21.15.81:0/2851736812 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fda7c016560 con 0x7fda840c5f70 2024-05-27T06:45:31.303 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:31.304 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-27T06:45:31.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 msgr2=0x7fda64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 0x7fda64071940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fda74005890 tx=0x7fda74009040 comp rx=0 tx=0).stop 2024-05-27T06:45:31.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 msgr2=0x7fda841f6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fda7c01a5a0 tx=0x7fda7c019710 comp rx=0 tx=0).stop 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 shutdown_connections 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fda840c5b60 0x7fda8414dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fda6406f430 0x7fda64071940 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda840c5f70 0x7fda841f6890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 --2- 172.21.15.81:0/2851736812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda84072430 0x7fda841f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 >> 172.21.15.81:0/2851736812 conn(0x7fda84070700 msgr2=0x7fda840749a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 shutdown_connections 2024-05-27T06:45:31.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.309+0000 7fda8ae00640 1 -- 172.21.15.81:0/2851736812 wait complete. 2024-05-27T06:45:31.325 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-27T06:45:31.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:31.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:45:31.462 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:45:31.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 Processor -- start 2024-05-27T06:45:31.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 -- start start 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb2e40ce220 0x7fb2e4153960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2e40c8160 con 0x7fb2e406ec30 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2e40c75e0 con 0x7fb2e415a000 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2ea800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb2e40c78c0 con 0x7fb2e40ce220 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb2e406ec30 0x7fb2e41598f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34594/0 (socket says 172.21.15.81:34594) 2024-05-27T06:45:31.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2a00640 1 -- 172.21.15.81:0/1940447850 learned_addr learned my addr 172.21.15.81:0/1940447850 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:31.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e3400640 1 --2- 172.21.15.81:0/1940447850 >> v2:172.21.15.106:3300/0 conn(0x7fb2e40ce220 0x7fb2e4153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980555205 0 0) 0x7fb2e40c8160 con 0x7fb2e406ec30 2024-05-27T06:45:31.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2c8005740 con 0x7fb2e406ec30 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853915864 0 0) 0x7fb2e40c75e0 con 0x7fb2e415a000 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2e40c8160 con 0x7fb2e415a000 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e3400640 1 -- 172.21.15.81:0/1940447850 >> v1:172.21.15.106:6789/0 conn(0x7fb2e415a000 legacy=0x7fb2e4163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e3400640 1 -- 172.21.15.81:0/1940447850 >> v1:172.21.15.81:6789/0 conn(0x7fb2e406ec30 legacy=0x7fb2e41598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 860164203 0 0) 0x7fb2c8005740 con 0x7fb2e406ec30 2024-05-27T06:45:31.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.481+0000 7fb2e3400640 1 -- 172.21.15.81:0/1940447850 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e40c7ba0 con 0x7fb2e40ce220 2024-05-27T06:45:31.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2e3400640 1 --2- 172.21.15.81:0/1940447850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e4153960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40097c0 tx=0x7fb2d401ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a975d6c348f89861 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40b5070 con 0x7fb2e40ce220 2024-05-27T06:45:31.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d40a8630 con 0x7fb2e40ce220 2024-05-27T06:45:31.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2e2000640 1 -- 172.21.15.81:0/1940447850 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0001880 con 0x7fb2e40ce220 2024-05-27T06:45:31.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 -- 172.21.15.81:0/1940447850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 msgr2=0x7fb2e4153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 --2- 172.21.15.81:0/1940447850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e4153960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2d40097c0 tx=0x7fb2d401ebb0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 -- 172.21.15.81:0/1940447850 shutdown_connections 2024-05-27T06:45:31.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 --2- 172.21.15.81:0/1940447850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e4153960 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 -- 172.21.15.81:0/1940447850 >> 172.21.15.81:0/1940447850 conn(0x7fb2e40cb2e0 msgr2=0x7fb2e40cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:31.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 -- 172.21.15.81:0/1940447850 shutdown_connections 2024-05-27T06:45:31.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.485+0000 7fb2ea800640 1 -- 172.21.15.81:0/1940447850 wait complete. 2024-05-27T06:45:31.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 Processor -- start 2024-05-27T06:45:31.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 -- start start 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e41f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2e415a000 0x7fb2e4146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e40c57c0 con 0x7fb2e406ec30 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e40c5d30 con 0x7fb2e40ce220 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2ea800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e40c5940 con 0x7fb2e415a000 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2e3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2e3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35134/0 (socket says 172.21.15.81:35134) 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2e3400640 1 -- 172.21.15.81:0/3511336649 learned_addr learned my addr 172.21.15.81:0/3511336649 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2e3e00640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2e415a000 0x7fb2e4146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.489+0000 7fb2e2a00640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e41f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 -- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2e415a000 msgr2=0x7fb2e4146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2e415a000 0x7fb2e4146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 -- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 msgr2=0x7fb2e41f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e41f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 -- 172.21.15.81:0/3511336649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e4074fa0 con 0x7fb2e406ec30 2024-05-27T06:45:31.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e2a00640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e41f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2e3400640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2d409b3d0 tx=0x7fb2d401b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40b5040 con 0x7fb2e406ec30 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d4004020 con 0x7fb2e406ec30 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4006e20 con 0x7fb2e406ec30 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e41f63c0 con 0x7fb2e406ec30 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e4148c20 con 0x7fb2e406ec30 2024-05-27T06:45:31.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e406a560 con 0x7fb2e406ec30 2024-05-27T06:45:31.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.493+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb2d40c2020 con 0x7fb2e406ec30 2024-05-27T06:45:31.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.497+0000 7fb2db400640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 0x7fb2c4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.497+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2d412bb30 con 0x7fb2e406ec30 2024-05-27T06:45:31.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.497+0000 7fb2e2a00640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 0x7fb2c4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.497+0000 7fb2e2a00640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 0x7fb2c4071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000f00 tx=0x7fb2d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.501+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb2d4018710 con 0x7fb2e406ec30 2024-05-27T06:45:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2e4159b40 con 0x7fb2e406ec30 2024-05-27T06:45:31.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2db400640 1 -- 172.21.15.81:0/3511336649 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6897 (secure 0 0 0) 0x7fb2d40f8aa0 con 0x7fb2e406ec30 2024-05-27T06:45:31.689 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:31.689 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:28.595118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:45:31.689 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-05-27T06:45:31.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 msgr2=0x7fb2c4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 0x7fb2c4071910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000f00 tx=0x7fb2d0009040 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 msgr2=0x7fb2e4074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2d409b3d0 tx=0x7fb2d401b950 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 shutdown_connections 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2e415a000 0x7fb2e4146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2c406f400 0x7fb2c4071910 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2e40ce220 0x7fb2e41f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 --2- 172.21.15.81:0/3511336649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406ec30 0x7fb2e4074610 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 >> 172.21.15.81:0/3511336649 conn(0x7fb2e40cb2e0 msgr2=0x7fb2e40cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 shutdown_connections 2024-05-27T06:45:31.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.693+0000 7fb2ea800640 1 -- 172.21.15.81:0/3511336649 wait complete. 2024-05-27T06:45:31.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:31.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 Processor -- start 2024-05-27T06:45:31.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 -- start start 2024-05-27T06:45:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ad415cb80 0x7f1ad4152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1ad406c790 0x7f1ad406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad40c9f80 con 0x7f1ad40c8880 2024-05-27T06:45:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad40c58b0 con 0x7f1ad415cb80 2024-05-27T06:45:31.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad9000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad40c83f0 con 0x7f1ad406c790 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ad415cb80 0x7f1ad4152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ad415cb80 0x7f1ad4152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35150/0 (socket says 172.21.15.81:35150) 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 -- 172.21.15.81:0/470070762 learned_addr learned my addr 172.21.15.81:0/470070762 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 -- 172.21.15.81:0/470070762 >> v2:172.21.15.106:3300/0 conn(0x7f1ad406c790 msgr2=0x7f1ad406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 --2- 172.21.15.81:0/470070762 >> v2:172.21.15.106:3300/0 conn(0x7f1ad406c790 0x7f1ad406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 -- 172.21.15.81:0/470070762 >> v1:172.21.15.106:6789/0 conn(0x7f1ad40c8880 legacy=0x7f1ad40c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:31.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.865+0000 7f1ad1e00640 1 -- 172.21.15.81:0/470070762 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad40c52f0 con 0x7f1ad415cb80 2024-05-27T06:45:31.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad1e00640 1 --2- 172.21.15.81:0/470070762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4152cb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4071360 tx=0x7f1ac401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877517bc130066d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad0a00640 1 -- 172.21.15.81:0/470070762 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40b0070 con 0x7f1ad415cb80 2024-05-27T06:45:31.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad0a00640 1 -- 172.21.15.81:0/470070762 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac40ab600 con 0x7f1ad415cb80 2024-05-27T06:45:31.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad0a00640 1 -- 172.21.15.81:0/470070762 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac40ab9d0 con 0x7f1ad415cb80 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 -- 172.21.15.81:0/470070762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 msgr2=0x7f1ad4152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 --2- 172.21.15.81:0/470070762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4152cb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4071360 tx=0x7f1ac401c520 comp rx=0 tx=0).stop 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 -- 172.21.15.81:0/470070762 shutdown_connections 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 --2- 172.21.15.81:0/470070762 >> v2:172.21.15.106:3300/0 conn(0x7f1ad406c790 0x7f1ad406cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 --2- 172.21.15.81:0/470070762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4152cb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 -- 172.21.15.81:0/470070762 >> 172.21.15.81:0/470070762 conn(0x7f1ad4070700 msgr2=0x7f1ad40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:31.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 -- 172.21.15.81:0/470070762 shutdown_connections 2024-05-27T06:45:31.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.869+0000 7f1ad9000640 1 -- 172.21.15.81:0/470070762 wait complete. 2024-05-27T06:45:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 Processor -- start 2024-05-27T06:45:31.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 -- start start 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad406c790 0x7f1ad4151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 0x7f1ad41644f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40c9c80 con 0x7f1ad415cb80 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad40c9b00 con 0x7f1ad40c8880 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad9000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad41634e0 con 0x7f1ad406c790 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35154/0 (socket says 172.21.15.81:35154) 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 -- 172.21.15.81:0/4288525436 learned_addr learned my addr 172.21.15.81:0/4288525436 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:31.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad1e00640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad406c790 0x7f1ad4151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad1400640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 0x7f1ad41644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 -- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad406c790 msgr2=0x7f1ad4151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad406c790 0x7f1ad4151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 -- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 msgr2=0x7f1ad41644f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 0x7f1ad41644f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 -- 172.21.15.81:0/4288525436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad414b1b0 con 0x7f1ad415cb80 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad1400640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 0x7f1ad41644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1ad2800640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00ba10 tx=0x7f1acc00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.873+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc019070 con 0x7f1ad415cb80 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1acc00cba0 con 0x7f1ad415cb80 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc014440 con 0x7f1ad415cb80 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad414b490 con 0x7f1ad415cb80 2024-05-27T06:45:31.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad415ac50 con 0x7f1ad415cb80 2024-05-27T06:45:31.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad406a560 con 0x7f1ad415cb80 2024-05-27T06:45:31.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1acc01c070 con 0x7f1ad415cb80 2024-05-27T06:45:31.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1aca000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 0x7f1ab4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:31.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1acc08efe0 con 0x7f1ad415cb80 2024-05-27T06:45:31.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.877+0000 7f1ad1e00640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 0x7f1ab4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:31.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.881+0000 7f1ad1e00640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 0x7f1ab4071910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ad000 tx=0x7f1ac40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:31.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:31.881+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1acc05b800 con 0x7f1ad415cb80 2024-05-27T06:45:32.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.073+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ad4075450 con 0x7f1ad415cb80 2024-05-27T06:45:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.073+0000 7f1aca000640 1 -- 172.21.15.81:0/4288525436 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6897 (secure 0 0 0) 0x7f1acc05fb00 con 0x7f1ad415cb80 2024-05-27T06:45:32.071 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:32.071 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:28.595118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:45:32.071 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 msgr2=0x7f1ab4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 0x7f1ab4071910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ad000 tx=0x7f1ac40a6000 comp rx=0 tx=0).stop 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 msgr2=0x7f1ad4164a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00ba10 tx=0x7f1acc00bed0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 shutdown_connections 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ab406f400 0x7f1ab4071910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ad415cb80 0x7f1ad4164a30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ad40c8880 0x7f1ad41644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 --2- 172.21.15.81:0/4288525436 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ad406c790 0x7f1ad4151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 >> 172.21.15.81:0/4288525436 conn(0x7f1ad4070700 msgr2=0x7f1ad40cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 shutdown_connections 2024-05-27T06:45:32.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.077+0000 7f1ad9000640 1 -- 172.21.15.81:0/4288525436 wait complete. 2024-05-27T06:45:32.092 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.241+0000 7f0ca3c00640 1 Processor -- start 2024-05-27T06:45:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.241+0000 7f0ca3c00640 1 -- start start 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca3c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 0x7f0c9c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca3c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 0x7f0c9c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca3c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9c072840 con 0x7f0c9c15a040 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca3c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9c0c5040 con 0x7f0c9c072430 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca3c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c0c75c0 con 0x7f0c9c161cd0 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 0x7f0c9c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 0x7f0c9c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0c93e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f0c9c15a040 0x7f0c9c0c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:59338/0 (socket says 172.21.15.81:59338) 2024-05-27T06:45:32.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0c93e00640 1 -- 172.21.15.81:0/1246179443 learned_addr learned my addr 172.21.15.81:0/1246179443 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:32.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0c9be00640 1 -- 172.21.15.81:0/1246179443 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2119089297 0 0) 0x7f0c9c072840 con 0x7f0c9c15a040 2024-05-27T06:45:32.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0c9be00640 1 -- 172.21.15.81:0/1246179443 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c80003530 con 0x7f0c9c15a040 2024-05-27T06:45:32.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1c00640 1 -- 172.21.15.81:0/1246179443 >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 msgr2=0x7f0c9c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:32.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1c00640 1 --2- 172.21.15.81:0/1246179443 >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 0x7f0c9c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1c00640 1 -- 172.21.15.81:0/1246179443 >> v1:172.21.15.106:6789/0 conn(0x7f0c9c15a040 legacy=0x7f0c9c0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1c00640 1 -- 172.21.15.81:0/1246179443 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c0c5270 con 0x7f0c9c161cd0 2024-05-27T06:45:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.245+0000 7f0ca1200640 1 --2- 172.21.15.81:0/1246179443 >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 0x7f0c9c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:32.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca1c00640 1 --2- 172.21.15.81:0/1246179443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 0x7f0c9c1640f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009d90 tx=0x7f0c9401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acc22ec7ce149c36 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0c9be00640 1 -- 172.21.15.81:0/1246179443 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b2070 con 0x7f0c9c161cd0 2024-05-27T06:45:32.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0c9be00640 1 -- 172.21.15.81:0/1246179443 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c940a5830 con 0x7f0c9c161cd0 2024-05-27T06:45:32.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0c9be00640 1 -- 172.21.15.81:0/1246179443 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940a5c00 con 0x7f0c9c161cd0 2024-05-27T06:45:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 -- 172.21.15.81:0/1246179443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 msgr2=0x7f0c9c1640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/1246179443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 0x7f0c9c1640f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c94009d90 tx=0x7f0c9401c970 comp rx=0 tx=0).stop 2024-05-27T06:45:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 -- 172.21.15.81:0/1246179443 shutdown_connections 2024-05-27T06:45:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/1246179443 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c161cd0 0x7f0c9c1640f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/1246179443 >> v2:172.21.15.106:3300/0 conn(0x7f0c9c072430 0x7f0c9c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 -- 172.21.15.81:0/1246179443 >> 172.21.15.81:0/1246179443 conn(0x7f0c9c070700 msgr2=0x7f0c9c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:32.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 -- 172.21.15.81:0/1246179443 shutdown_connections 2024-05-27T06:45:32.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.249+0000 7f0ca3c00640 1 -- 172.21.15.81:0/1246179443 wait complete. 2024-05-27T06:45:32.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 Processor -- start 2024-05-27T06:45:32.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 -- start start 2024-05-27T06:45:32.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c072430 0x7f0c9c14f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c9c15a040 0x7f0c9c14b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c0ce6f0 con 0x7f0c9c161cd0 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9c0c7dd0 con 0x7f0c9c15a040 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca3c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9c0c5610 con 0x7f0c9c072430 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35168/0 (socket says 172.21.15.81:35168) 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 -- 172.21.15.81:0/2162967246 learned_addr learned my addr 172.21.15.81:0/2162967246 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1200640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c072430 0x7f0c9c14f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0c93e00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c9c15a040 0x7f0c9c14b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 -- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c072430 msgr2=0x7f0c9c14f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c072430 0x7f0c9c14f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 -- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c9c15a040 msgr2=0x7f0c9c14b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c9c15a040 0x7f0c9c14b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.253+0000 7f0ca1c00640 1 -- 172.21.15.81:0/2162967246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9c14ca90 con 0x7f0c9c161cd0 2024-05-27T06:45:32.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0ca1c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0c940b0040 tx=0x7f0c94019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b2050 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c940be070 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9c15c500 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940ad890 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9c14e190 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c9c14e360 con 0x7f0c9c161cd0 2024-05-27T06:45:32.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0c940ada30 con 0x7f0c9c161cd0 2024-05-27T06:45:32.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 0x7f0c78071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0c9412a1b0 con 0x7f0c9c161cd0 2024-05-27T06:45:32.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.257+0000 7f0ca1200640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 0x7f0c78071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.261+0000 7f0ca1200640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 0x7f0c78071910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009580 tx=0x7f0c8c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.265+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0c94016560 con 0x7f0c9c161cd0 2024-05-27T06:45:32.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c9c06a560 con 0x7f0c9c161cd0 2024-05-27T06:45:32.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0c99600640 1 -- 172.21.15.81:0/2162967246 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f0c940f6a40 con 0x7f0c9c161cd0 2024-05-27T06:45:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:32.425 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:03.608328+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 msgr2=0x7f0c78071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 0x7f0c78071910 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009580 tx=0x7f0c8c009190 comp rx=0 tx=0).stop 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 msgr2=0x7f0c9c14b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0c940b0040 tx=0x7f0c94019710 comp rx=0 tx=0).stop 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 shutdown_connections 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c7806f400 0x7f0c78071910 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c9c161cd0 0x7f0c9c14b710 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c9c15a040 0x7f0c9c14b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 --2- 172.21.15.81:0/2162967246 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c9c072430 0x7f0c9c14f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 >> 172.21.15.81:0/2162967246 conn(0x7f0c9c070700 msgr2=0x7f0c9c0c8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:32.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 shutdown_connections 2024-05-27T06:45:32.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.429+0000 7f0ca3c00640 1 -- 172.21.15.81:0/2162967246 wait complete. 2024-05-27T06:45:32.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:32.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.597+0000 7fc19a400640 1 Processor -- start 2024-05-27T06:45:32.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.597+0000 7fc19a400640 1 -- start start 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc19a400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc1941546f0 0x7fc194072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc19a400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc19415db00 con 0x7fc1940732f0 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc19a400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc194073860 con 0x7fc19415a510 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc19a400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc19415e6f0 con 0x7fc1941546f0 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc18aa00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc1940732f0 0x7fc1940c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34600/0 (socket says 172.21.15.81:34600) 2024-05-27T06:45:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc18aa00640 1 -- 172.21.15.81:0/4007719441 learned_addr learned my addr 172.21.15.81:0/4007719441 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193400640 1 --2- 172.21.15.81:0/4007719441 >> v2:172.21.15.106:3300/0 conn(0x7fc1941546f0 0x7fc194072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472586634 0 0) 0x7fc19415db00 con 0x7fc1940732f0 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc178003530 con 0x7fc1940732f0 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249280539 0 0) 0x7fc194073860 con 0x7fc19415a510 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc19415db00 con 0x7fc19415a510 2024-05-27T06:45:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2665829431 0 0) 0x7fc178003530 con 0x7fc1940732f0 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 >> v2:172.21.15.106:3300/0 conn(0x7fc1941546f0 msgr2=0x7fc194072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 --2- 172.21.15.81:0/4007719441 >> v2:172.21.15.106:3300/0 conn(0x7fc1941546f0 0x7fc194072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 >> v1:172.21.15.106:6789/0 conn(0x7fc19415a510 legacy=0x7fc19415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19415ddc0 con 0x7fc1940732f0 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fc17c001700 con 0x7fc1940732f0 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193400640 1 --2- 172.21.15.81:0/4007719441 >> v2:172.21.15.106:3300/0 conn(0x7fc1941546f0 0x7fc194072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 >> v1:172.21.15.81:6789/0 conn(0x7fc1940732f0 legacy=0x7fc1940c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178003c40 0x7fc178004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc178004810 0x7fc178004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 0x7fc17800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415e6f0 con 0x7fc178005290 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc17800c1a0 con 0x7fc178003c40 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc17800c3a0 con 0x7fc178004810 2024-05-27T06:45:32.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 0x7fc17800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193400640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178003c40 0x7fc178004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc18aa00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc178004810 0x7fc178004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 -- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc178004810 msgr2=0x7fc178004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc178004810 0x7fc178004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 -- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178003c40 msgr2=0x7fc178004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178003c40 0x7fc178004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.601+0000 7fc193e00640 1 -- 172.21.15.81:0/4007719441 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc178003530 con 0x7fc178005290 2024-05-27T06:45:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc193e00640 1 -- 172.21.15.81:0/4007719441 reap_dead start 2024-05-27T06:45:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc193e00640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 0x7fc17800b9d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1940ce6e0 tx=0x7fc18c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597377dff1bcbcf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c001c50 con 0x7fc178005290 2024-05-27T06:45:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18c003b80 con 0x7fc178005290 2024-05-27T06:45:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc192a00640 1 -- 172.21.15.81:0/4007719441 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c0050d0 con 0x7fc178005290 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 -- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 msgr2=0x7fc17800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 0x7fc17800b9d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1940ce6e0 tx=0x7fc18c01c8c0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 -- 172.21.15.81:0/4007719441 shutdown_connections 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 --2- 172.21.15.81:0/4007719441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc178005290 0x7fc17800b9d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 -- 172.21.15.81:0/4007719441 >> 172.21.15.81:0/4007719441 conn(0x7fc194070700 msgr2=0x7fc1940cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 -- 172.21.15.81:0/4007719441 shutdown_connections 2024-05-27T06:45:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.605+0000 7fc19a400640 1 -- 172.21.15.81:0/4007719441 wait complete. 2024-05-27T06:45:32.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 Processor -- start 2024-05-27T06:45:32.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 -- start start 2024-05-27T06:45:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178005290 0x7fc1941f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 0x7fc19414c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940755b0 con 0x7fc194152f80 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415e380 con 0x7fc178005290 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc19a400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc194154b00 con 0x7fc1940cbbf0 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35178/0 (socket says 172.21.15.81:35178) 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 -- 172.21.15.81:0/1960877104 learned_addr learned my addr 172.21.15.81:0/1960877104 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc193400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178005290 0x7fc1941f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc193e00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 0x7fc19414c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 -- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 msgr2=0x7fc19414c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 0x7fc19414c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 -- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178005290 msgr2=0x7fc1941f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178005290 0x7fc1941f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 -- 172.21.15.81:0/1960877104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1941f4080 con 0x7fc194152f80 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc193e00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 0x7fc19414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:32.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18aa00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0072c0 tx=0x7fc17c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00f9c0 con 0x7fc194152f80 2024-05-27T06:45:32.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.609+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17c018070 con 0x7fc194152f80 2024-05-27T06:45:32.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1941537c0 con 0x7fc194152f80 2024-05-27T06:45:32.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00fda0 con 0x7fc194152f80 2024-05-27T06:45:32.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1941f3340 con 0x7fc194152f80 2024-05-27T06:45:32.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc194153ce0 con 0x7fc194152f80 2024-05-27T06:45:32.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc17c01c410 con 0x7fc194152f80 2024-05-27T06:45:32.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc18be00640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 0x7fc170071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:32.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc17c08ef10 con 0x7fc194152f80 2024-05-27T06:45:32.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc193400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 0x7fc170071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:32.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.613+0000 7fc193400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 0x7fc170071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1840093f0 tx=0x7fc184009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:32.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.617+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc17c05be90 con 0x7fc194152f80 2024-05-27T06:45:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.809+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1941f3130 con 0x7fc194152f80 2024-05-27T06:45:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.809+0000 7fc18be00640 1 -- 172.21.15.81:0/1960877104 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6897 (secure 0 0 0) 0x7fc17c05b790 con 0x7fc194152f80 2024-05-27T06:45:32.805 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:32.805 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:28.595118+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:creating","state_seq":1,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:45:32.806 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 16 2024-05-27T06:45:32.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 msgr2=0x7fc170071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 0x7fc170071940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc1840093f0 tx=0x7fc184009190 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 msgr2=0x7fc19414c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0072c0 tx=0x7fc17c007780 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 shutdown_connections 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc1940cbbf0 0x7fc19414c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc17006f430 0x7fc170071940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc194152f80 0x7fc19414c050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 --2- 172.21.15.81:0/1960877104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc178005290 0x7fc1941f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 >> 172.21.15.81:0/1960877104 conn(0x7fc194070700 msgr2=0x7fc1940c6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 shutdown_connections 2024-05-27T06:45:32.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:32.813+0000 7fc19a400640 1 -- 172.21.15.81:0/1960877104 wait complete. 2024-05-27T06:45:33.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T06:45:02.016832+0000', 'modified': '2024-05-27T06:45:28.595118+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4302, 'mds_1': 4287, 'mds_2': 4285, 'mds_3': 4281, 'mds_4': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.106:6833/353223641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 353223641}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 353223641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4281': {'gid': 4281, 'name': 'c', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.81:6835/2990302599', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2990302599}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 2990302599}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4285': {'gid': 4285, 'name': 'b', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.106:6837/13170464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6835', 'nonce': 13170464}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 13170464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4287': {'gid': 4287, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.81:6839/328699254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6837', 'nonce': 328699254}, {'type': 'v1', 'addr': '172.21.15.81:6839', 'nonce': 328699254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4302': {'gid': 4302, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.106:6836/730205609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 730205609}, {'type': 'v1', 'addr': '172.21.15.106:6836', 'nonce': 730205609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4302, 'qdb_cluster': [4302, 4287, 4285, 4281]} 2024-05-27T06:45:33.431 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4279:up:creating 2024-05-27T06:45:34.432 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:34.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 Processor -- start 2024-05-27T06:45:34.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 -- start start 2024-05-27T06:45:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f83800ca1e0 0x7f8380154740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 0x7f838015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838015b520 con 0x7f8380072090 2024-05-27T06:45:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f83800c9e20 con 0x7f83800ca1e0 2024-05-27T06:45:34.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.597+0000 7f8385400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f838015b090 con 0x7f838015dbb0 2024-05-27T06:45:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837f200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8380072090 0x7f8380074630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34614/0 (socket says 172.21.15.81:34614) 2024-05-27T06:45:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f83800ca1e0 0x7f8380154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837f200640 1 -- 172.21.15.81:0/1068092447 learned_addr learned my addr 172.21.15.81:0/1068092447 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837de00640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 0x7f838015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837d400640 1 -- 172.21.15.81:0/1068092447 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2148522667 0 0) 0x7f838015b520 con 0x7f8380072090 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837d400640 1 -- 172.21.15.81:0/1068092447 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8364003530 con 0x7f8380072090 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 -- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 msgr2=0x7f838015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 0x7f838015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 -- 172.21.15.81:0/1068092447 >> v1:172.21.15.81:6789/0 conn(0x7f8380072090 legacy=0x7f8380074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 -- 172.21.15.81:0/1068092447 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83800c9b40 con 0x7f83800ca1e0 2024-05-27T06:45:34.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837de00640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 0x7f838015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.601+0000 7f837e800640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83800ca1e0 0x7f8380154740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f83700097f0 tx=0x7f837001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de13c4bbcf6686a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f837d400640 1 -- 172.21.15.81:0/1068092447 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700b5070 con 0x7f83800ca1e0 2024-05-27T06:45:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f837d400640 1 -- 172.21.15.81:0/1068092447 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83700a8990 con 0x7f83800ca1e0 2024-05-27T06:45:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f837d400640 1 -- 172.21.15.81:0/1068092447 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700a8d60 con 0x7f83800ca1e0 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83800ca1e0 msgr2=0x7f8380154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83800ca1e0 0x7f8380154740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f83700097f0 tx=0x7f837001ec90 comp rx=0 tx=0).stop 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- 172.21.15.81:0/1068092447 shutdown_connections 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f838015dbb0 0x7f838015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 --2- 172.21.15.81:0/1068092447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83800ca1e0 0x7f8380154740 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- 172.21.15.81:0/1068092447 >> 172.21.15.81:0/1068092447 conn(0x7f8380068cb0 msgr2=0x7f83800cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:34.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- 172.21.15.81:0/1068092447 shutdown_connections 2024-05-27T06:45:34.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- 172.21.15.81:0/1068092447 wait complete. 2024-05-27T06:45:34.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 Processor -- start 2024-05-27T06:45:34.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.605+0000 7f8385400640 1 -- start start 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 0x7f8380157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f838015dbb0 0x7f83800c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f838015ad20 con 0x7f83800ca1e0 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f838015aa40 con 0x7f838015dbb0 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f83800d07a0 con 0x7f8380072090 2024-05-27T06:45:34.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35186/0 (socket says 172.21.15.81:35186) 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 -- 172.21.15.81:0/1608612478 learned_addr learned my addr 172.21.15.81:0/1608612478 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837e800640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 0x7f8380157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837f200640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f838015dbb0 0x7f83800c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 -- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 msgr2=0x7f8380157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 0x7f8380157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 -- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f838015dbb0 msgr2=0x7f83800c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f838015dbb0 0x7f83800c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 -- 172.21.15.81:0/1608612478 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83801577a0 con 0x7f83800ca1e0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837e800640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 0x7f8380157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f837de00640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f836c00e9b0 tx=0x7f836c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c017070 con 0x7f83800ca1e0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f836c00ca00 con 0x7f83800ca1e0 2024-05-27T06:45:34.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836c00cdd0 con 0x7f83800ca1e0 2024-05-27T06:45:34.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83801f40e0 con 0x7f83800ca1e0 2024-05-27T06:45:34.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83801f4680 con 0x7f83800ca1e0 2024-05-27T06:45:34.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.609+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f838006a560 con 0x7f83800ca1e0 2024-05-27T06:45:34.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.613+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f836c01a070 con 0x7f83800ca1e0 2024-05-27T06:45:34.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.613+0000 7f8376a00640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 0x7f8360071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:34.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.613+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f836c08f240 con 0x7f83800ca1e0 2024-05-27T06:45:34.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.613+0000 7f837e800640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 0x7f8360071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:34.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.613+0000 7f837e800640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 0x7f8360071910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f83700b2000 tx=0x7f8370008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:34.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.617+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f836c05c0d0 con 0x7f83800ca1e0 2024-05-27T06:45:34.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.809+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83800ca840 con 0x7f83800ca1e0 2024-05-27T06:45:34.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8376a00640 1 -- 172.21.15.81:0/1608612478 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f836c05b9d0 con 0x7f83800ca1e0 2024-05-27T06:45:34.808 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:34.808 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:45:34.808 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 msgr2=0x7f8360071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 0x7f8360071910 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f83700b2000 tx=0x7f8370008040 comp rx=0 tx=0).stop 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 msgr2=0x7f838015c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f836c00e9b0 tx=0x7f836c00ee70 comp rx=0 tx=0).stop 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 shutdown_connections 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f836006f400 0x7f8360071910 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f838015dbb0 0x7f83800c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83800ca1e0 0x7f838015c400 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 --2- 172.21.15.81:0/1608612478 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8380072090 0x7f8380157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:34.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 >> 172.21.15.81:0/1608612478 conn(0x7f8380068cb0 msgr2=0x7f83801591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:34.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 shutdown_connections 2024-05-27T06:45:34.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:34.813+0000 7f8385400640 1 -- 172.21.15.81:0/1608612478 wait complete. 2024-05-27T06:45:35.433 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-27T06:45:02.016832+0000', 'modified': '2024-05-27T06:45:34.378415+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4302, 'mds_1': 4287, 'mds_2': 4285, 'mds_3': 4281, 'mds_4': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.106:6833/353223641', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 353223641}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 353223641}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4281': {'gid': 4281, 'name': 'c', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.81:6835/2990302599', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2990302599}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 2990302599}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4285': {'gid': 4285, 'name': 'b', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.106:6837/13170464', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6835', 'nonce': 13170464}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 13170464}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4287': {'gid': 4287, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.81:6839/328699254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6837', 'nonce': 328699254}, {'type': 'v1', 'addr': '172.21.15.81:6839', 'nonce': 328699254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4302': {'gid': 4302, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.106:6836/730205609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 730205609}, {'type': 'v1', 'addr': '172.21.15.106:6836', 'nonce': 730205609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4302, 'qdb_cluster': [4279, 4281, 4285, 4287, 4302]} 2024-05-27T06:45:35.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-27T06:45:35.434 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-27T06:45:35.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.589+0000 7efe36600640 1 Processor -- start 2024-05-27T06:45:35.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.589+0000 7efe36600640 1 -- start start 2024-05-27T06:45:35.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe36600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efe30160680 0x7efe300c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe36600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3015ce70 con 0x7efe3015c750 2024-05-27T06:45:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe36600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3015d150 con 0x7efe300cedd0 2024-05-27T06:45:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe36600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7efe3015d710 con 0x7efe30160680 2024-05-27T06:45:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2f400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efe3015c750 0x7efe301569a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60442/0 (socket says 172.21.15.81:60442) 2024-05-27T06:45:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2f400640 1 -- 172.21.15.81:0/4110693731 learned_addr learned my addr 172.21.15.81:0/4110693731 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:35.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2ea00640 1 --2- 172.21.15.81:0/4110693731 >> v2:172.21.15.106:3300/0 conn(0x7efe30160680 0x7efe300c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750459692 0 0) 0x7efe3015ce70 con 0x7efe3015c750 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe14003530 con 0x7efe3015c750 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470361505 0 0) 0x7efe3015d150 con 0x7efe300cedd0 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe3015ce70 con 0x7efe300cedd0 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1436526350 0 0) 0x7efe14003530 con 0x7efe3015c750 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 >> v2:172.21.15.106:3300/0 conn(0x7efe30160680 msgr2=0x7efe300c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 --2- 172.21.15.81:0/4110693731 >> v2:172.21.15.106:3300/0 conn(0x7efe30160680 0x7efe300c92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 >> v1:172.21.15.106:6789/0 conn(0x7efe300cedd0 legacy=0x7efe30167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3015d430 con 0x7efe3015c750 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7efe20001700 con 0x7efe3015c750 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 >> v1:172.21.15.81:6789/0 conn(0x7efe3015c750 legacy=0x7efe301569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe14003c40 0x7efe14004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14004810 0x7efe14004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe3015d710 con 0x7efe14005290 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efe1400c1a0 con 0x7efe14003c40 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efe1400c3a0 con 0x7efe14004810 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2f400640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe14003c40 0x7efe14004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2ea00640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14004810 0x7efe14004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 -- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14004810 msgr2=0x7efe14004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe14004810 0x7efe14004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 -- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe14003c40 msgr2=0x7efe14004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe14003c40 0x7efe14004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.593+0000 7efe2fe00640 1 -- 172.21.15.81:0/4110693731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe14003530 con 0x7efe14005290 2024-05-27T06:45:35.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe2fe00640 1 -- 172.21.15.81:0/4110693731 reap_dead start 2024-05-27T06:45:35.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe2fe00640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe1400b9d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe28002080 tx=0x7efe2801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4ef2a41f3b63d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:35.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28001c50 con 0x7efe14005290 2024-05-27T06:45:35.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe28003b80 con 0x7efe14005290 2024-05-27T06:45:35.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe2e000640 1 -- 172.21.15.81:0/4110693731 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe280050d0 con 0x7efe14005290 2024-05-27T06:45:35.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 -- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 msgr2=0x7efe1400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe1400b9d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe28002080 tx=0x7efe2801c900 comp rx=0 tx=0).stop 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 -- 172.21.15.81:0/4110693731 shutdown_connections 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 --2- 172.21.15.81:0/4110693731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe1400b9d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 -- 172.21.15.81:0/4110693731 >> 172.21.15.81:0/4110693731 conn(0x7efe3006dca0 msgr2=0x7efe3006e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 -- 172.21.15.81:0/4110693731 shutdown_connections 2024-05-27T06:45:35.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.597+0000 7efe36600640 1 -- 172.21.15.81:0/4110693731 wait complete. 2024-05-27T06:45:35.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 Processor -- start 2024-05-27T06:45:35.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- start start 2024-05-27T06:45:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe3015f950 0x7efe301f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe300c63e0 0x7efe300cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe3015cb60 con 0x7efe14005290 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efe30160160 con 0x7efe3015f950 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efe301e8fe0 con 0x7efe300c63e0 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41426/0 (socket says 172.21.15.81:41426) 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 -- 172.21.15.81:0/75026555 learned_addr learned my addr 172.21.15.81:0/75026555 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2ea00640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe3015f950 0x7efe301f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2fe00640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe300c63e0 0x7efe300cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 -- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe300c63e0 msgr2=0x7efe300cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe300c63e0 0x7efe300cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 -- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe3015f950 msgr2=0x7efe301f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe3015f950 0x7efe301f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 -- 172.21.15.81:0/75026555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe301543a0 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe2f400640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe20004f80 tx=0x7efe20007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe27400640 1 -- 172.21.15.81:0/75026555 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2001a070 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe27400640 1 -- 172.21.15.81:0/75026555 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe2000f7b0 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe300c8650 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe27400640 1 -- 172.21.15.81:0/75026555 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2000fba0 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.601+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe301f2ea0 con 0x7efe14005290 2024-05-27T06:45:35.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe27400640 1 -- 172.21.15.81:0/75026555 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efe2000fd40 con 0x7efe14005290 2024-05-27T06:45:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe27400640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 0x7efe10071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe27400640 1 -- 172.21.15.81:0/75026555 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efe2008ead0 con 0x7efe14005290 2024-05-27T06:45:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe2ea00640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 0x7efe10071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe3dc32000 1 Processor -- start 2024-05-27T06:45:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe3dc32000 1 -- start start 2024-05-27T06:45:35.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.605+0000 7efe2ea00640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 0x7efe10071940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe30151f80 tx=0x7efe1c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d931f0 0x558f78e9afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78ed3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558f78cef6f0 con 0x558f78d8bf90 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558f78cdf290 con 0x558f78d75a60 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x558f78ce32d0 con 0x558f78d931f0 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41436/0 (socket says 172.21.15.81:41436) 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 -- 172.21.15.81:0/796394643 learned_addr learned my addr 172.21.15.81:0/796394643 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2fe00640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78ed3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2f400640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d931f0 0x558f78e9afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 -- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d931f0 msgr2=0x558f78e9afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d931f0 0x558f78e9afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 -- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 msgr2=0x558f78ed3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78ed3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 -- 172.21.15.81:0/796394643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f78cbdc10 con 0x558f78d8bf90 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2fe00640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78ed3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:35.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe2ea00640 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efe1c014f80 tx=0x7efe1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af463c679b392d65 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:35.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe1be00640 1 -- 172.21.15.81:0/796394643 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1c01b070 con 0x558f78d8bf90 2024-05-27T06:45:35.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe1be00640 1 -- 172.21.15.81:0/796394643 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c019070 con 0x558f78d8bf90 2024-05-27T06:45:35.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe1be00640 1 -- 172.21.15.81:0/796394643 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1c015420 con 0x558f78d8bf90 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- 172.21.15.81:0/796394643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 msgr2=0x558f78e9b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efe1c014f80 tx=0x7efe1c017000 comp rx=0 tx=0).stop 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- 172.21.15.81:0/796394643 shutdown_connections 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78ed3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d8bf90 0x558f78e9b500 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 --2- 172.21.15.81:0/796394643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d931f0 0x558f78e9afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- 172.21.15.81:0/796394643 >> 172.21.15.81:0/796394643 conn(0x558f78d99db0 msgr2=0x558f78ed1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- 172.21.15.81:0/796394643 shutdown_connections 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.609+0000 7efe3dc32000 1 -- 172.21.15.81:0/796394643 wait complete. 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- start start 2024-05-27T06:45:35.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 Processor -- start 2024-05-27T06:45:35.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e94670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e94bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558f78ca2650 con 0x558f78d931f0 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558f78d63590 con 0x558f78d75a60 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x558f78cee4c0 con 0x558f78d8bf90 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 10 client.0 _ll_register_callbacks cb 0x558f78e7e560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41448/0 (socket says 172.21.15.81:41448) 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2f400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e94670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 -- 172.21.15.81:0/2807196688 learned_addr learned my addr 172.21.15.81:0/2807196688 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2ea00640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e94bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 -- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 msgr2=0x558f78e94bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e94bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 -- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 msgr2=0x558f78e94670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e94670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 -- 172.21.15.81:0/2807196688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f78d279d0 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe2fe00640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe28002080 tx=0x7efe28019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9645ed7ab8befc3e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe18c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe280bc060 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe1c019070 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efe280054e0 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7efe280c2050 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 10 client.4632 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x558f78cba530 con 0x558f78d931f0 2024-05-27T06:45:35.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.613+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c019070 con 0x558f78d931f0 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7efe280c5020 con 0x558f78d931f0 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 10 client.4632 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 10 client.4632 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 20 client.4632 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f789877d0 0x558f78e95b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:35.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 4 client.4632 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T06:45:35.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 --> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f78c7abe0 con 0x558f789877d0 2024-05-27T06:45:35.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe2f400640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f789877d0 0x558f78e95b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:35.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe18c00640 10 client.4632 ms_handle_connect on v2:172.21.15.106:6832/353223641 2024-05-27T06:45:35.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:35.617+0000 7efe2f400640 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f789877d0 0x558f78e95b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e334301b61b773fe server_cookie=a9bf6acf84ffc9ca in_seq=0 out_seq=0 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe18c00640 1 -- 172.21.15.81:0/2807196688 <== mds.4 v2:172.21.15.106:6832/353223641 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x558f78c7abe0 con 0x558f789877d0 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe18c00640 10 client.4632 handle_command_reply: tid=0 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 client.4632 shutdown 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 msgr2=0x558f78e950f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efe28002080 tx=0x7efe28019740 comp rx=0 tx=0).stop 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 shutdown_connections 2024-05-27T06:45:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f789877d0 0x558f78e95b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e94bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78e950f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 --2- 172.21.15.81:0/2807196688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e94670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 >> 172.21.15.81:0/2807196688 conn(0x558f78d99db0 msgr2=0x558f78ed1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe18c00640 0 client.4632 ms_handle_reset on v2:172.21.15.106:6832/353223641 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 shutdown_connections 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 1 -- 172.21.15.81:0/2807196688 wait complete. 2024-05-27T06:45:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.389+0000 7efe3dc32000 20 client.4632 trim_cache size 0 max 16384 2024-05-27T06:45:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.405+0000 7efe3dc32000 1 Processor -- start 2024-05-27T06:45:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.405+0000 7efe3dc32000 1 -- start start 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e99b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e9a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe301ffd30 con 0x558f78d931f0 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efe301f6a50 con 0x558f78d75a60 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x558f78c7c220 con 0x558f78d8bf90 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41456/0 (socket says 172.21.15.81:41456) 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 -- 172.21.15.81:0/3413235472 learned_addr learned my addr 172.21.15.81:0/3413235472 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2f400640 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e99b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2ea00640 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e9a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 -- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 msgr2=0x558f78e9a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e9a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 -- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 msgr2=0x558f78e99b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e99b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 -- 172.21.15.81:0/3413235472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f78c80880 con 0x558f78d931f0 2024-05-27T06:45:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe2fe00640 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe2801b000 tx=0x7efe280b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabd9be181974704 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe1b400640 1 -- 172.21.15.81:0/3413235472 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28005820 con 0x558f78d931f0 2024-05-27T06:45:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe1b400640 1 -- 172.21.15.81:0/3413235472 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe280052e0 con 0x558f78d931f0 2024-05-27T06:45:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe1b400640 1 -- 172.21.15.81:0/3413235472 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe280ad790 con 0x558f78d931f0 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- 172.21.15.81:0/3413235472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 msgr2=0x558f78f44230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efe2801b000 tx=0x7efe280b4000 comp rx=0 tx=0).stop 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- 172.21.15.81:0/3413235472 shutdown_connections 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d8bf90 0x558f78e9a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f44230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 --2- 172.21.15.81:0/3413235472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d75a60 0x558f78e99b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- 172.21.15.81:0/3413235472 >> 172.21.15.81:0/3413235472 conn(0x558f78d99db0 msgr2=0x558f78eee620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- 172.21.15.81:0/3413235472 shutdown_connections 2024-05-27T06:45:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.409+0000 7efe3dc32000 1 -- 172.21.15.81:0/3413235472 wait complete. 2024-05-27T06:45:39.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 -- start start 2024-05-27T06:45:39.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 Processor -- start 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d75a60 0x558f78ed4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 0x558f78f48270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558f78ce25e0 con 0x558f78d931f0 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558f78c777d0 con 0x558f78d8bf90 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x558f78ce7900 con 0x558f78d75a60 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 10 client.0 _ll_register_callbacks cb 0x7efe14003890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41468/0 (socket says 172.21.15.81:41468) 2024-05-27T06:45:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 -- 172.21.15.81:0/555852659 learned_addr learned my addr 172.21.15.81:0/555852659 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2f400640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d75a60 0x558f78ed4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2ea00640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 0x558f78f48270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 -- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d75a60 msgr2=0x558f78ed4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d75a60 0x558f78ed4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 -- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 msgr2=0x558f78f48270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 0x558f78f48270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 -- 172.21.15.81:0/555852659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f78c96ea0 con 0x558f78d931f0 2024-05-27T06:45:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2ea00640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 0x558f78f48270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe2fe00640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efe28001f50 tx=0x7efe280a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d356a0586b33ec4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe18c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe280ad790 con 0x558f78d931f0 2024-05-27T06:45:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe280052e0 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efe280ad930 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.413+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7efe280d4050 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 10 client.4638 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x558f78d63a20 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe280052e0 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7efe280c5030 con 0x558f78d931f0 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 10 client.4638 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 10 client.4638 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 20 client.4638 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f78975830 0x558f78f47050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 4 client.4638 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 --> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] -- command(tid 0: {"prefix": "status"}) -- 0x558f78c7ede0 con 0x558f78975830 2024-05-27T06:45:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe2f400640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f78975830 0x558f78f47050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 10 client.4638 ms_handle_connect on v2:172.21.15.106:6832/353223641 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe2f400640 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f78975830 0x558f78f47050 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1243dbb5010b75b5 server_cookie=2c7b4c4973ae5aac in_seq=0 out_seq=0 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 1 -- 172.21.15.81:0/555852659 <== mds.4 v2:172.21.15.106:6832/353223641 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558f78c7ede0 con 0x558f78975830 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 10 client.4638 handle_command_reply: tid=0 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 client.4638 shutdown 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 msgr2=0x558f78f487b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efe28001f50 tx=0x7efe280a7070 comp rx=0 tx=0).stop 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 shutdown_connections 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:6832/353223641,v1:172.21.15.106:6833/353223641] conn(0x558f78975830 0x558f78f47050 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f78d931f0 0x558f78f487b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558f78d8bf90 0x558f78f48270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 --2- 172.21.15.81:0/555852659 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x558f78d75a60 0x558f78ed4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 >> 172.21.15.81:0/555852659 conn(0x558f78d99db0 msgr2=0x558f78e9af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe18c00640 0 client.4638 ms_handle_reset on v2:172.21.15.106:6832/353223641 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 shutdown_connections 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 1 -- 172.21.15.81:0/555852659 wait complete. 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe3dc32000 20 client.4638 trim_cache size 0 max 16384 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "cb957b26-d515-43f2-962a-6a97b6477eea", 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 4, 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4279, 2024-05-27T06:45:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 10.823631885999999, 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 21, 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 21, 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 42.424807833999999 2024-05-27T06:45:39.417 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 msgr2=0x7efe10071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 0x7efe10071940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe30151f80 tx=0x7efe1c009190 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 msgr2=0x7efe300c7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe20004f80 tx=0x7efe20007430 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 shutdown_connections 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efe300c63e0 0x7efe300cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efe1006f430 0x7efe10071940 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efe3015f950 0x7efe301f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 --2- 172.21.15.81:0/75026555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe14005290 0x7efe300c7d30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 >> 172.21.15.81:0/75026555 conn(0x7efe3006dca0 msgr2=0x7efe3015b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 shutdown_connections 2024-05-27T06:45:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.417+0000 7efe36600640 1 -- 172.21.15.81:0/75026555 wait complete. 2024-05-27T06:45:39.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-27T06:45:39.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.597+0000 7ff4cd200640 1 Processor -- start 2024-05-27T06:45:39.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.597+0000 7ff4cd200640 1 -- start start 2024-05-27T06:45:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4cd200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff4c80c4d20 0x7ff4c8153940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4cd200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073460 0x7ff4c8163eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4cd200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c80c66a0 con 0x7ff4c8073050 2024-05-27T06:45:39.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4cd200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c80c5130 con 0x7ff4c80c4d20 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4cd200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c815a210 con 0x7ff4c8073460 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff4c80c4d20 0x7ff4c8153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff4c8073050 0x7ff4c80c7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60448/0 (socket says 172.21.15.81:60448) 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5400640 1 -- 172.21.15.81:0/345584804 learned_addr learned my addr 172.21.15.81:0/345584804 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c6800640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073460 0x7ff4c8163eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5e00640 1 -- 172.21.15.81:0/345584804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073460 msgr2=0x7ff4c8163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073460 0x7ff4c8163eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5e00640 1 -- 172.21.15.81:0/345584804 >> v1:172.21.15.81:6789/0 conn(0x7ff4c8073050 legacy=0x7ff4c80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c5e00640 1 -- 172.21.15.81:0/345584804 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c80c6940 con 0x7ff4c80c4d20 2024-05-27T06:45:39.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.601+0000 7ff4c4a00640 1 -- 172.21.15.81:0/345584804 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218364594 0 0) 0x7ff4c80c66a0 con 0x7ff4c8073050 2024-05-27T06:45:39.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c8153940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097c0 tx=0x7ff4b801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5df4062507dcbb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4c4a00640 1 -- 172.21.15.81:0/345584804 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80a7070 con 0x7ff4c80c4d20 2024-05-27T06:45:39.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4c4a00640 1 -- 172.21.15.81:0/345584804 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b80b0070 con 0x7ff4c80c4d20 2024-05-27T06:45:39.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4c4a00640 1 -- 172.21.15.81:0/345584804 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80ab890 con 0x7ff4c80c4d20 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 -- 172.21.15.81:0/345584804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 msgr2=0x7ff4c8153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c8153940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097c0 tx=0x7ff4b801c470 comp rx=0 tx=0).stop 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 -- 172.21.15.81:0/345584804 shutdown_connections 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073460 0x7ff4c8163eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 --2- 172.21.15.81:0/345584804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c8153940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 -- 172.21.15.81:0/345584804 >> 172.21.15.81:0/345584804 conn(0x7ff4c806cd90 msgr2=0x7ff4c80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 -- 172.21.15.81:0/345584804 shutdown_connections 2024-05-27T06:45:39.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 -- 172.21.15.81:0/345584804 wait complete. 2024-05-27T06:45:39.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.605+0000 7ff4cd200640 1 Processor -- start 2024-05-27T06:45:39.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- start start 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 0x7ff4c8143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4c8073460 0x7ff4c814d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c815a4f0 con 0x7ff4c80c4d20 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c8073870 con 0x7ff4c8073460 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c8072b20 con 0x7ff4c8073050 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41476/0 (socket says 172.21.15.81:41476) 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 -- 172.21.15.81:0/4102460989 learned_addr learned my addr 172.21.15.81:0/4102460989 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 0x7ff4c8143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c5400640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4c8073460 0x7ff4c814d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 -- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 msgr2=0x7ff4c8143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 0x7ff4c8143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 -- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4c8073460 msgr2=0x7ff4c814d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4c8073460 0x7ff4c814d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 -- 172.21.15.81:0/4102460989 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c8153230 con 0x7ff4c80c4d20 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 0x7ff4c8143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4c6800640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000efc0 tx=0x7ff4c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4be000640 1 -- 172.21.15.81:0/4102460989 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0018020 con 0x7ff4c80c4d20 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4be000640 1 -- 172.21.15.81:0/4102460989 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c0014480 con 0x7ff4c80c4d20 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4be000640 1 -- 172.21.15.81:0/4102460989 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0009e60 con 0x7ff4c80c4d20 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c81641e0 con 0x7ff4c80c4d20 2024-05-27T06:45:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.609+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c8141ab0 con 0x7ff4c80c4d20 2024-05-27T06:45:39.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4be000640 1 -- 172.21.15.81:0/4102460989 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff4c001b020 con 0x7ff4c80c4d20 2024-05-27T06:45:39.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4be000640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 0x7ff4a8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4be000640 1 -- 172.21.15.81:0/4102460989 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff4c008ffa0 con 0x7ff4c80c4d20 2024-05-27T06:45:39.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 0x7ff4a8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4d492d000 1 Processor -- start 2024-05-27T06:45:39.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.613+0000 7ff4d492d000 1 -- start start 2024-05-27T06:45:39.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 0x7ff4a8071910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80af000 tx=0x7ff4b80a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b649c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b647e36f0 con 0x561b6487ff90 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561b647d3290 con 0x561b648871f0 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x561b647d72d0 con 0x561b64869a60 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41492/0 (socket says 172.21.15.81:41492) 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 -- 172.21.15.81:0/2386389677 learned_addr learned my addr 172.21.15.81:0/2386389677 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c6800640 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b649c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 -- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 msgr2=0x561b649c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b649c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 -- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 msgr2=0x561b6498efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 -- 172.21.15.81:0/2386389677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b647b1c10 con 0x561b6487ff90 2024-05-27T06:45:39.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4c5400640 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40074d0 tx=0x7ff4b4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ffbb0d80a4a8c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4b2a00640 1 -- 172.21.15.81:0/2386389677 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4019070 con 0x561b6487ff90 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4b2a00640 1 -- 172.21.15.81:0/2386389677 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4001740 con 0x561b6487ff90 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4b2a00640 1 -- 172.21.15.81:0/2386389677 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4001b10 con 0x561b6487ff90 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- 172.21.15.81:0/2386389677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 msgr2=0x561b6498f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40074d0 tx=0x7ff4b4005bd0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- 172.21.15.81:0/2386389677 shutdown_connections 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b649c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498f500 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2386389677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- 172.21.15.81:0/2386389677 >> 172.21.15.81:0/2386389677 conn(0x561b6488ddb0 msgr2=0x561b649c5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- 172.21.15.81:0/2386389677 shutdown_connections 2024-05-27T06:45:39.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.617+0000 7ff4d492d000 1 -- 172.21.15.81:0/2386389677 wait complete. 2024-05-27T06:45:39.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- start start 2024-05-27T06:45:39.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 Processor -- start 2024-05-27T06:45:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b64988670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b6487ff90 0x561b64988bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b64796650 con 0x561b648871f0 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561b64857590 con 0x561b6487ff90 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x561b647e24c0 con 0x561b64869a60 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 10 client.0 _ll_register_callbacks cb 0x561b64972560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41506/0 (socket says 172.21.15.81:41506) 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 -- 172.21.15.81:0/941470666 learned_addr learned my addr 172.21.15.81:0/941470666 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 -- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 msgr2=0x561b64988670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b64988670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 -- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b6487ff90 msgr2=0x561b64988bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b6487ff90 0x561b64988bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 -- 172.21.15.81:0/941470666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6481b9d0 con 0x561b648871f0 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c6800640 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0058fd0 tx=0x7ff4c005c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2e033410f3261 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c008c060 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff4c005db40 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b4001740 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff4c007a050 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 10 client.4653 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x561b647ae530 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4001740 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff4c005ba70 con 0x561b648871f0 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 10 client.4653 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 10 client.4653 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 20 client.4653 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b6447b7d0 0x561b64989b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 4 client.4653 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T06:45:39.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 --> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b6476ebe0 con 0x561b6447b7d0 2024-05-27T06:45:39.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.621+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b6447b7d0 0x561b64989b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4a7400640 10 client.4653 ms_handle_connect on v2:172.21.15.81:6834/2990302599 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b6447b7d0 0x561b64989b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ac345ff1943f0e23 server_cookie=49e3778cc1eec680 in_seq=0 out_seq=0 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4a7400640 1 -- 172.21.15.81:0/941470666 <== mds.3 v2:172.21.15.81:6834/2990302599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x561b6476ebe0 con 0x561b6447b7d0 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4a7400640 10 client.4653 handle_command_reply: tid=0 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 client.4653 shutdown 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 msgr2=0x561b649890f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0058fd0 tx=0x7ff4c005c3e0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 shutdown_connections 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b6447b7d0 0x561b64989b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b649890f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b6487ff90 0x561b64988bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 --2- 172.21.15.81:0/941470666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b64988670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 >> 172.21.15.81:0/941470666 conn(0x561b6488ddb0 msgr2=0x561b649c5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4a7400640 0 client.4653 ms_handle_reset on v2:172.21.15.81:6834/2990302599 2024-05-27T06:45:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 shutdown_connections 2024-05-27T06:45:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 1 -- 172.21.15.81:0/941470666 wait complete. 2024-05-27T06:45:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.625+0000 7ff4d492d000 20 client.4653 trim_cache size 0 max 16384 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 Processor -- start 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- start start 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b6498e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c8202230 con 0x561b6487ff90 2024-05-27T06:45:39.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c81517a0 con 0x561b648871f0 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x561b64770220 con 0x561b64869a60 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41510/0 (socket says 172.21.15.81:41510) 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 -- 172.21.15.81:0/3550405841 learned_addr learned my addr 172.21.15.81:0/3550405841 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b6498e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c6800640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 -- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 msgr2=0x561b6498e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b6498e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 -- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 msgr2=0x561b6498e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 -- 172.21.15.81:0/3550405841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b64774880 con 0x561b6487ff90 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c6800640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:39.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4c5400640 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x561b64967400 tx=0x7ff4b4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969dad00391da9af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4b2000640 1 -- 172.21.15.81:0/3550405841 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4023070 con 0x561b6487ff90 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4b2000640 1 -- 172.21.15.81:0/3550405841 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b401a040 con 0x561b6487ff90 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4b2000640 1 -- 172.21.15.81:0/3550405841 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4019030 con 0x561b6487ff90 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- 172.21.15.81:0/3550405841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 msgr2=0x561b6498db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x561b64967400 tx=0x7ff4b4011000 comp rx=0 tx=0).stop 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- 172.21.15.81:0/3550405841 shutdown_connections 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b648871f0 0x561b6498e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b6487ff90 0x561b6498db20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 --2- 172.21.15.81:0/3550405841 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b64869a60 0x561b6498e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- 172.21.15.81:0/3550405841 >> 172.21.15.81:0/3550405841 conn(0x561b6488ddb0 msgr2=0x561b649e2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- 172.21.15.81:0/3550405841 shutdown_connections 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.645+0000 7ff4d492d000 1 -- 172.21.15.81:0/3550405841 wait complete. 2024-05-27T06:45:39.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- start start 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 Processor -- start 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b64869a60 0x561b64a3ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 0x561b64a34fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b647d65e0 con 0x561b648871f0 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561b6476b7d0 con 0x561b64869a60 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x561b647db900 con 0x561b6487ff90 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 10 client.0 _ll_register_callbacks cb 0x7ff4c005c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41522/0 (socket says 172.21.15.81:41522) 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 -- 172.21.15.81:0/2839505435 learned_addr learned my addr 172.21.15.81:0/2839505435 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c5400640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 0x561b64a34fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b64869a60 0x561b64a3ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 -- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 msgr2=0x561b64a34fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 0x561b64a34fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 -- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b64869a60 msgr2=0x561b64a3ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b64869a60 0x561b64a3ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 -- 172.21.15.81:0/2839505435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b6478aea0 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c5400640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 0x561b64a34fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4c6800640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0058e00 tx=0x7ff4c008e390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04b1c5de53d0f32 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0075070 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff4c005db40 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b401a040 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff4c0060bc0 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 10 client.4659 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x561b64857a20 con 0x561b648871f0 2024-05-27T06:45:39.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b401a040 con 0x561b648871f0 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.649+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff4c0058180 con 0x561b648871f0 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 10 client.4659 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 10 client.4659 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 20 client.4659 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b64469830 0x561b64a3b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 4 client.4659 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 --> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] -- command(tid 0: {"prefix": "status"}) -- 0x561b64772de0 con 0x561b64469830 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b64469830 0x561b64a3b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4c5e00640 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b64469830 0x561b64a3b610 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=56b630c8cba254a5 server_cookie=b4804a28d30b3131 in_seq=0 out_seq=0 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4a7400640 10 client.4659 ms_handle_connect on v2:172.21.15.81:6834/2990302599 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4a7400640 1 -- 172.21.15.81:0/2839505435 <== mds.3 v2:172.21.15.81:6834/2990302599 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x561b64772de0 con 0x561b64469830 2024-05-27T06:45:39.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4a7400640 10 client.4659 handle_command_reply: tid=0 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 client.4659 shutdown 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 msgr2=0x561b64a35520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0058e00 tx=0x7ff4c008e390 comp rx=0 tx=0).stop 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 shutdown_connections 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:6834/2990302599,v1:172.21.15.81:6835/2990302599] conn(0x561b64469830 0x561b64a3b610 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x561b6487ff90 0x561b64a34fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b648871f0 0x561b64a35520 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 --2- 172.21.15.81:0/2839505435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b64869a60 0x561b64a3ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4a7400640 0 client.4659 ms_handle_reset on v2:172.21.15.81:6834/2990302599 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 >> 172.21.15.81:0/2839505435 conn(0x561b6488ddb0 msgr2=0x561b649d8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 shutdown_connections 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 1 -- 172.21.15.81:0/2839505435 wait complete. 2024-05-27T06:45:39.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4d492d000 20 client.4659 trim_cache size 0 max 16384 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "cb957b26-d515-43f2-962a-6a97b6477eea", 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 3, 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4281, 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 16.449778938000001, 2024-05-27T06:45:39.656 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-05-27T06:45:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 21, 2024-05-27T06:45:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 21, 2024-05-27T06:45:39.657 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 42.677506276999999 2024-05-27T06:45:39.657 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 msgr2=0x7ff4a8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 0x7ff4a8071910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80af000 tx=0x7ff4b80a7000 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 msgr2=0x7ff4c814d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000efc0 tx=0x7ff4c000c490 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 shutdown_connections 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff4a806f400 0x7ff4a8071910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4c80c4d20 0x7ff4c814d5d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4c8073460 0x7ff4c814d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 --2- 172.21.15.81:0/4102460989 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff4c8073050 0x7ff4c8143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 >> 172.21.15.81:0/4102460989 conn(0x7ff4c806cd90 msgr2=0x7ff4c80c82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 shutdown_connections 2024-05-27T06:45:39.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.653+0000 7ff4cd200640 1 -- 172.21.15.81:0/4102460989 wait complete. 2024-05-27T06:45:39.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-27T06:45:39.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.821+0000 7f889cc00640 1 Processor -- start 2024-05-27T06:45:39.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.821+0000 7f889cc00640 1 -- start start 2024-05-27T06:45:39.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f889cc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 0x7f889815df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f889cc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88980c7d40 con 0x7f8898154530 2024-05-27T06:45:39.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f889cc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88980c89f0 con 0x7f8898074650 2024-05-27T06:45:39.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f889cc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f889815afe0 con 0x7f889815db40 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8898154530 0x7f88980c7630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60458/0 (socket says 172.21.15.81:60458) 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895e00640 1 -- 172.21.15.81:0/2162180080 learned_addr learned my addr 172.21.15.81:0/2162180080 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895400640 1 --2- 172.21.15.81:0/2162180080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 0x7f889815df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251763083 0 0) 0x7f88980c7d40 con 0x7f8898154530 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f887c003530 con 0x7f8898154530 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934561371 0 0) 0x7f88980c89f0 con 0x7f8898074650 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88980c7d40 con 0x7f8898074650 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895400640 1 -- 172.21.15.81:0/2162180080 >> v1:172.21.15.106:6789/0 conn(0x7f8898074650 legacy=0x7f8898164ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895400640 1 -- 172.21.15.81:0/2162180080 >> v1:172.21.15.81:6789/0 conn(0x7f8898154530 legacy=0x7f88980c7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.825+0000 7f8895400640 1 -- 172.21.15.81:0/2162180080 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889815a6f0 con 0x7f889815db40 2024-05-27T06:45:39.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f8895400640 1 --2- 172.21.15.81:0/2162180080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 0x7f889815df50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88840097f0 tx=0x7f888401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f871dd1d1513925c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840b5070 con 0x7f889815db40 2024-05-27T06:45:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840a89d0 con 0x7f889815db40 2024-05-27T06:45:39.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f8894a00640 1 -- 172.21.15.81:0/2162180080 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840a8da0 con 0x7f889815db40 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 -- 172.21.15.81:0/2162180080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 msgr2=0x7f889815df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 --2- 172.21.15.81:0/2162180080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 0x7f889815df50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f88840097f0 tx=0x7f888401ecd0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 -- 172.21.15.81:0/2162180080 shutdown_connections 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 --2- 172.21.15.81:0/2162180080 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f889815db40 0x7f889815df50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 -- 172.21.15.81:0/2162180080 >> 172.21.15.81:0/2162180080 conn(0x7f889806cd90 msgr2=0x7f88980cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 -- 172.21.15.81:0/2162180080 shutdown_connections 2024-05-27T06:45:39.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.829+0000 7f889cc00640 1 -- 172.21.15.81:0/2162180080 wait complete. 2024-05-27T06:45:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 Processor -- start 2024-05-27T06:45:39.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- start start 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 0x7f88981e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 0x7f88981e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f889815a990 con 0x7f8898074650 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f88980c8710 con 0x7f889815db40 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f889815ac70 con 0x7f8898154530 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41536/0 (socket says 172.21.15.81:41536) 2024-05-27T06:45:39.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 -- 172.21.15.81:0/85341324 learned_addr learned my addr 172.21.15.81:0/85341324 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895400640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 0x7f88981e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8896800640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 0x7f88981e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 -- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 msgr2=0x7f88981e35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 0x7f88981e35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 -- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 msgr2=0x7f88981e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 0x7f88981e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 -- 172.21.15.81:0/85341324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88980c6570 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895400640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 0x7f88981e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8896800640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 0x7f88981e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f8895e00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f888800c480 tx=0x7f888800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f888e000640 1 -- 172.21.15.81:0/85341324 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888017070 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f888e000640 1 -- 172.21.15.81:0/85341324 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8888005cf0 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f888e000640 1 -- 172.21.15.81:0/85341324 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8888007270 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88981e6430 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.833+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88981e6150 con 0x7f8898074650 2024-05-27T06:45:39.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f888e000640 1 -- 172.21.15.81:0/85341324 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f888801a070 con 0x7f8898074650 2024-05-27T06:45:39.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f888e000640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 0x7f8878071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f888e000640 1 -- 172.21.15.81:0/85341324 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f888808e4d0 con 0x7f8898074650 2024-05-27T06:45:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f8895400640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 0x7f8878071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f88a42e5000 1 Processor -- start 2024-05-27T06:45:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f88a42e5000 1 -- start start 2024-05-27T06:45:39.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.837+0000 7f8895400640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 0x7f8878071940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f88840b2000 tx=0x7f88840aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 0x55e1a42adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 0x55e1a42e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1a41026f0 con 0x55e1a419ef90 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e1a40f2290 con 0x55e1a41a61f0 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55e1a40f62d0 con 0x55e1a4188a60 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41552/0 (socket says 172.21.15.81:41552) 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 -- 172.21.15.81:0/4160859 learned_addr learned my addr 172.21.15.81:0/4160859 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895e00640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 0x55e1a42adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 -- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 msgr2=0x55e1a42e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8896800640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 0x55e1a42e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 0x55e1a42e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 -- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 msgr2=0x55e1a42adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 0x55e1a42adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 -- 172.21.15.81:0/4160859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1a40d0c10 con 0x55e1a419ef90 2024-05-27T06:45:39.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895e00640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 0x55e1a42adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:39.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8896800640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 0x55e1a42e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:39.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8895400640 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f888401c900 tx=0x7f8884004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a795a4d004b4721 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8882600640 1 -- 172.21.15.81:0/4160859 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884004450 con 0x55e1a419ef90 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8882600640 1 -- 172.21.15.81:0/4160859 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840b6040 con 0x55e1a419ef90 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f8882600640 1 -- 172.21.15.81:0/4160859 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840a89d0 con 0x55e1a419ef90 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- 172.21.15.81:0/4160859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 msgr2=0x55e1a42ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f888401c900 tx=0x7f8884004740 comp rx=0 tx=0).stop 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- 172.21.15.81:0/4160859 shutdown_connections 2024-05-27T06:45:39.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a4188a60 0x55e1a42e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42ae500 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 --2- 172.21.15.81:0/4160859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41a61f0 0x55e1a42adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- 172.21.15.81:0/4160859 >> 172.21.15.81:0/4160859 conn(0x55e1a41acdb0 msgr2=0x55e1a42e4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- 172.21.15.81:0/4160859 shutdown_connections 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.841+0000 7f88a42e5000 1 -- 172.21.15.81:0/4160859 wait complete. 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- start start 2024-05-27T06:45:39.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 Processor -- start 2024-05-27T06:45:39.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a4188a60 0x55e1a42a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a41a61f0 0x55e1a42a80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1a40b5650 con 0x55e1a419ef90 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e1a4176590 con 0x55e1a4188a60 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55e1a41014c0 con 0x55e1a41a61f0 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 10 client.0 _ll_register_callbacks cb 0x55e1a4291560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41568/0 (socket says 172.21.15.81:41568) 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 -- 172.21.15.81:0/869419944 learned_addr learned my addr 172.21.15.81:0/869419944 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8896800640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a41a61f0 0x55e1a42a80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895e00640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a4188a60 0x55e1a42a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 -- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a41a61f0 msgr2=0x55e1a42a80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a41a61f0 0x55e1a42a80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 -- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a4188a60 msgr2=0x55e1a42a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:39.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a4188a60 0x55e1a42a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:39.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 -- 172.21.15.81:0/869419944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1a413a9d0 con 0x55e1a419ef90 2024-05-27T06:45:39.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8895400640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f88840098e0 tx=0x7f888401d510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0645bde58082bf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:39.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840a8ca0 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f88840cfac0 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88840b6040 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f88840d6050 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 10 client.4671 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55e1a40cd530 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840b6040 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f88840d9020 con 0x55e1a419ef90 2024-05-27T06:45:39.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 10 client.4671 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 10 client.4671 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 20 client.4671 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3d9a7d0 0x55e1a42fba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 4 client.4671 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.845+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 --> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1a408dbe0 con 0x55e1a3d9a7d0 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.849+0000 7f8895e00640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3d9a7d0 0x55e1a42fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:39.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.849+0000 7f8876a00640 10 client.4671 ms_handle_connect on v2:172.21.15.106:6835/13170464 2024-05-27T06:45:39.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:39.849+0000 7f8895e00640 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3d9a7d0 0x55e1a42fba70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8d1e0b252a915de server_cookie=6fce392b58df8401 in_seq=0 out_seq=0 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f8876a00640 1 -- 172.21.15.81:0/869419944 <== mds.2 v2:172.21.15.106:6835/13170464 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55e1a408dbe0 con 0x55e1a3d9a7d0 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f8876a00640 10 client.4671 handle_command_reply: tid=0 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 client.4671 shutdown 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 msgr2=0x55e1a42a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f88840098e0 tx=0x7f888401d510 comp rx=0 tx=0).stop 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 shutdown_connections 2024-05-27T06:45:40.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3d9a7d0 0x55e1a42fba70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a41a61f0 0x55e1a42a80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a419ef90 0x55e1a42a7bb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f8876a00640 0 client.4671 ms_handle_reset on v2:172.21.15.106:6835/13170464 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 --2- 172.21.15.81:0/869419944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a4188a60 0x55e1a42a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 >> 172.21.15.81:0/869419944 conn(0x55e1a41acdb0 msgr2=0x55e1a42e4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 shutdown_connections 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 1 -- 172.21.15.81:0/869419944 wait complete. 2024-05-27T06:45:40.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.693+0000 7f88a42e5000 20 client.4671 trim_cache size 0 max 16384 2024-05-27T06:45:40.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.713+0000 7f88a42e5000 1 Processor -- start 2024-05-27T06:45:40.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.713+0000 7f88a42e5000 1 -- start start 2024-05-27T06:45:40.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a4354410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a4354e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f88981ff0b0 con 0x55e1a4188a60 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f88981ff780 con 0x55e1a41acdb0 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f889814e520 con 0x55e1a419ef90 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41580/0 (socket says 172.21.15.81:41580) 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 -- 172.21.15.81:0/2857083626 learned_addr learned my addr 172.21.15.81:0/2857083626 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8896800640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a4354e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895e00640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a4354410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 -- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 msgr2=0x55e1a4354e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a4354e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 -- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 msgr2=0x55e1a4354410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a4354410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 -- 172.21.15.81:0/2857083626 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1a4093880 con 0x55e1a4188a60 2024-05-27T06:45:40.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895e00640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a4354410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:40.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8896800640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a4354e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8895400640 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f888401f570 tx=0x7f888400b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8951039e267cd6a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8881c00640 1 -- 172.21.15.81:0/2857083626 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840a9c30 con 0x55e1a4188a60 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8881c00640 1 -- 172.21.15.81:0/2857083626 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840b6070 con 0x55e1a4188a60 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f8881c00640 1 -- 172.21.15.81:0/2857083626 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884003620 con 0x55e1a4188a60 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- 172.21.15.81:0/2857083626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 msgr2=0x55e1a4354950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f888401f570 tx=0x7f888400b050 comp rx=0 tx=0).stop 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- 172.21.15.81:0/2857083626 shutdown_connections 2024-05-27T06:45:40.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a4354e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a4354950 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 --2- 172.21.15.81:0/2857083626 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a4354410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- 172.21.15.81:0/2857083626 >> 172.21.15.81:0/2857083626 conn(0x55e1a42f7350 msgr2=0x55e1a42adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- 172.21.15.81:0/2857083626 shutdown_connections 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- 172.21.15.81:0/2857083626 wait complete. 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 -- start start 2024-05-27T06:45:40.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.717+0000 7f88a42e5000 1 Processor -- start 2024-05-27T06:45:40.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a435e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a431d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1a408a7d0 con 0x55e1a4188a60 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e1a408f220 con 0x55e1a41acdb0 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55e1a40f55e0 con 0x55e1a419ef90 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 10 client.0 _ll_register_callbacks cb 0x7f887c003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41586/0 (socket says 172.21.15.81:41586) 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 -- 172.21.15.81:0/3114335062 learned_addr learned my addr 172.21.15.81:0/3114335062 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895400640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a435e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8896800640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a431d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 -- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 msgr2=0x55e1a435e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a435e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 -- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 msgr2=0x55e1a431d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a431d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 -- 172.21.15.81:0/3114335062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1a40a9ea0 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8895e00640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8888058140 tx=0x7f8888063bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c6789db34a0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888808a060 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f888805a130 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88840b6070 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f888805a330 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 10 client.4677 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55e1a4176a20 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840b6070 con 0x55e1a4188a60 2024-05-27T06:45:40.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f888807bb70 con 0x55e1a4188a60 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 10 client.4677 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 10 client.4677 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 20 client.4677 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3db3e80 0x55e1a431dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 4 client.4677 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.721+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 --> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] -- command(tid 0: {"prefix": "status"}) -- 0x55e1a4091de0 con 0x55e1a3db3e80 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8895400640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3db3e80 0x55e1a431dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8895400640 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3db3e80 0x55e1a431dbd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59308bf01fa5e71b server_cookie=56f7e6f67eda468f in_seq=0 out_seq=0 2024-05-27T06:45:40.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8876a00640 10 client.4677 ms_handle_connect on v2:172.21.15.106:6835/13170464 2024-05-27T06:45:40.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8876a00640 1 -- 172.21.15.81:0/3114335062 <== mds.2 v2:172.21.15.106:6835/13170464 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55e1a4091de0 con 0x55e1a3db3e80 2024-05-27T06:45:40.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8876a00640 10 client.4677 handle_command_reply: tid=0 2024-05-27T06:45:40.722 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 client.4677 shutdown 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 msgr2=0x55e1a435e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8888058140 tx=0x7f8888063bf0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 shutdown_connections 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:6835/13170464,v1:172.21.15.106:6837/13170464] conn(0x55e1a3db3e80 0x55e1a431dbd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55e1a419ef90 0x55e1a435e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e1a41acdb0 0x55e1a431d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 --2- 172.21.15.81:0/3114335062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1a4188a60 0x55e1a435e350 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f8876a00640 0 client.4677 ms_handle_reset on v2:172.21.15.106:6835/13170464 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 >> 172.21.15.81:0/3114335062 conn(0x55e1a42f7350 msgr2=0x55e1a42adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 shutdown_connections 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 1 -- 172.21.15.81:0/3114335062 wait complete. 2024-05-27T06:45:40.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f88a42e5000 20 client.4677 trim_cache size 0 max 16384 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "cb957b26-d515-43f2-962a-6a97b6477eea", 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 2, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4285, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 23.855563555, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 21, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 21, 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 43.727814926000001 2024-05-27T06:45:40.724 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 msgr2=0x7f8878071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 0x7f8878071940 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f88840b2000 tx=0x7f88840aa070 comp rx=0 tx=0).stop 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 msgr2=0x7f8898150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f888800c480 tx=0x7f888800c940 comp rx=0 tx=0).stop 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 shutdown_connections 2024-05-27T06:45:40.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8898154530 0x7f88981e35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f887806f430 0x7f8878071940 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f889815db40 0x7f88981e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 --2- 172.21.15.81:0/85341324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898074650 0x7f8898150730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 >> 172.21.15.81:0/85341324 conn(0x7f889806cd90 msgr2=0x7f8898159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 shutdown_connections 2024-05-27T06:45:40.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.725+0000 7f889cc00640 1 -- 172.21.15.81:0/85341324 wait complete. 2024-05-27T06:45:40.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-27T06:45:40.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.897+0000 7ffbbe800640 1 Processor -- start 2024-05-27T06:45:40.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.897+0000 7ffbbe800640 1 -- start start 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbbe800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffbb80727a0 0x7ffbb8070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbbe800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffbb8152a50 0x7ffbb8152e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbbe800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb8073280 con 0x7ffbb80c4600 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbbe800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb80ca090 con 0x7ffbb8152a50 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbbe800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb80c9850 con 0x7ffbb80727a0 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffbb80c4600 0x7ffbb815c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60462/0 (socket says 172.21.15.81:60462) 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7400640 1 -- 172.21.15.81:0/1662630218 learned_addr learned my addr 172.21.15.81:0/1662630218 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1662630218 >> v2:172.21.15.81:3300/0 conn(0x7ffbb8152a50 0x7ffbb8152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1662630218 >> v2:172.21.15.106:3300/0 conn(0x7ffbb80727a0 0x7ffbb8070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb6000640 1 -- 172.21.15.81:0/1662630218 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3999722305 0 0) 0x7ffbb8073280 con 0x7ffbb80c4600 2024-05-27T06:45:40.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1662630218 >> v2:172.21.15.106:3300/0 conn(0x7ffbb80727a0 msgr2=0x7ffbb8070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1662630218 >> v2:172.21.15.106:3300/0 conn(0x7ffbb80727a0 0x7ffbb8070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1662630218 >> v1:172.21.15.81:6789/0 conn(0x7ffbb80c4600 legacy=0x7ffbb815c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1662630218 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb80c9ad0 con 0x7ffbb8152a50 2024-05-27T06:45:40.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.901+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1662630218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb8152a50 0x7ffbb8152e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009d90 tx=0x7ffbb001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5f307f1558d4aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbb6000640 1 -- 172.21.15.81:0/1662630218 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00b2070 con 0x7ffbb8152a50 2024-05-27T06:45:40.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbb6000640 1 -- 172.21.15.81:0/1662630218 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb00a5860 con 0x7ffbb8152a50 2024-05-27T06:45:40.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbb6000640 1 -- 172.21.15.81:0/1662630218 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00a5c30 con 0x7ffbb8152a50 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- 172.21.15.81:0/1662630218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb8152a50 msgr2=0x7ffbb8152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 --2- 172.21.15.81:0/1662630218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb8152a50 0x7ffbb8152e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0009d90 tx=0x7ffbb001c930 comp rx=0 tx=0).stop 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- 172.21.15.81:0/1662630218 shutdown_connections 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 --2- 172.21.15.81:0/1662630218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb8152a50 0x7ffbb8152e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 --2- 172.21.15.81:0/1662630218 >> v2:172.21.15.106:3300/0 conn(0x7ffbb80727a0 0x7ffbb8070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- 172.21.15.81:0/1662630218 >> 172.21.15.81:0/1662630218 conn(0x7ffbb806cd90 msgr2=0x7ffbb80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- 172.21.15.81:0/1662630218 shutdown_connections 2024-05-27T06:45:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- 172.21.15.81:0/1662630218 wait complete. 2024-05-27T06:45:40.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 Processor -- start 2024-05-27T06:45:40.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.905+0000 7ffbbe800640 1 -- start start 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80c4600 0x7ffbb81556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 0x7ffbb8153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81533a0 con 0x7ffbb80727a0 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8070570 con 0x7ffbb80c4600 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb8072bb0 con 0x7ffbb8152a50 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41602/0 (socket says 172.21.15.81:41602) 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 -- 172.21.15.81:0/2783723724 learned_addr learned my addr 172.21.15.81:0/2783723724 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 0x7ffbb8153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80c4600 0x7ffbb81556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 -- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 msgr2=0x7ffbb8153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 0x7ffbb8153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 -- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80c4600 msgr2=0x7ffbb81556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80c4600 0x7ffbb81556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 -- 172.21.15.81:0/2783723724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb80cc3c0 con 0x7ffbb80727a0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 0x7ffbb8153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbb7400640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffba800c430 tx=0x7ffba800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbaf400640 1 -- 172.21.15.81:0/2783723724 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba80074e0 con 0x7ffbb80727a0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbaf400640 1 -- 172.21.15.81:0/2783723724 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba80047f0 con 0x7ffbb80727a0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbaf400640 1 -- 172.21.15.81:0/2783723724 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8004be0 con 0x7ffbb80727a0 2024-05-27T06:45:40.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb81545b0 con 0x7ffbb80727a0 2024-05-27T06:45:40.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.909+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb8154b70 con 0x7ffbb80727a0 2024-05-27T06:45:40.907 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbaf400640 1 -- 172.21.15.81:0/2783723724 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffba8011070 con 0x7ffbb80727a0 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbaf400640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 0x7ffb94071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbaf400640 1 -- 172.21.15.81:0/2783723724 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffba808eda0 con 0x7ffbb80727a0 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 0x7ffb94071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 Processor -- start 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 -- start start 2024-05-27T06:45:40.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 0x7ffb94071910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cbaa0 tx=0x7ffba0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911dd1f0 0x55ac912e4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac9131dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ac911396f0 con 0x55ac911d5f90 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ac91129290 con 0x55ac911dd1f0 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.913+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55ac9112d2d0 con 0x55ac911bfa60 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac9131dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41606/0 (socket says 172.21.15.81:41606) 2024-05-27T06:45:40.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 -- 172.21.15.81:0/1801160990 learned_addr learned my addr 172.21.15.81:0/1801160990 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:40.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911dd1f0 0x55ac912e4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 -- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 msgr2=0x55ac9131dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac9131dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 -- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911dd1f0 msgr2=0x55ac912e4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911dd1f0 0x55ac912e4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 -- 172.21.15.81:0/1801160990 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac91107c10 con 0x55ac911d5f90 2024-05-27T06:45:40.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffba0011c80 tx=0x7ffba0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d1174b255d7733 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffba7e00640 1 -- 172.21.15.81:0/1801160990 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011cb0 con 0x55ac911d5f90 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffba7e00640 1 -- 172.21.15.81:0/1801160990 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba0026070 con 0x55ac911d5f90 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffba7e00640 1 -- 172.21.15.81:0/1801160990 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0021660 con 0x55ac911d5f90 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1801160990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 msgr2=0x55ac912e5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffba0011c80 tx=0x7ffba0014000 comp rx=0 tx=0).stop 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1801160990 shutdown_connections 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac9131dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911d5f90 0x55ac912e5500 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1801160990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911dd1f0 0x55ac912e4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1801160990 >> 172.21.15.81:0/1801160990 conn(0x55ac911e3db0 msgr2=0x55ac9131b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1801160990 shutdown_connections 2024-05-27T06:45:40.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1801160990 wait complete. 2024-05-27T06:45:40.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 -- start start 2024-05-27T06:45:40.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.917+0000 7ffbc5eb4000 1 Processor -- start 2024-05-27T06:45:40.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac912de630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912deb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ac910ec650 con 0x55ac911dd1f0 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ac911ad590 con 0x55ac911d5f90 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55ac911384c0 con 0x55ac911bfa60 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 10 client.0 _ll_register_callbacks cb 0x55ac912c8560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41618/0 (socket says 172.21.15.81:41618) 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1375341066 learned_addr learned my addr 172.21.15.81:0/1375341066 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac912de630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912deb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 msgr2=0x55ac912de630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac912de630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 msgr2=0x55ac912deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912deb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:40.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 -- 172.21.15.81:0/1375341066 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac911719d0 con 0x55ac911dd1f0 2024-05-27T06:45:40.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00b0040 tx=0x7ffbb001b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953b1e11637c0a2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:40.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00b2040 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba0026070 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffbb00c4ae0 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ffbb00c4ce0 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 10 client.4692 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55ac91104530 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba0026070 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ffbb00c4740 con 0x55ac911dd1f0 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 10 client.4692 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:40.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 10 client.4692 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 20 client.4692 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dd17d0 0x55ac912dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 4 client.4692 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 --> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac910c4be0 con 0x55ac90dd17d0 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dd17d0 0x55ac912dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dd17d0 0x55ac912dfac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eca7d6b94b68815c server_cookie=5790f3987c91a63 in_seq=0 out_seq=0 2024-05-27T06:45:40.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:40.921+0000 7ffba4c00640 10 client.4692 ms_handle_connect on v2:172.21.15.81:6837/328699254 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffba4c00640 1 -- 172.21.15.81:0/1375341066 <== mds.1 v2:172.21.15.81:6837/328699254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x55ac910c4be0 con 0x55ac90dd17d0 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffba4c00640 10 client.4692 handle_command_reply: tid=0 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 client.4692 shutdown 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 msgr2=0x55ac912df0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00b0040 tx=0x7ffbb001b170 comp rx=0 tx=0).stop 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 shutdown_connections 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dd17d0 0x55ac912dfac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911dd1f0 0x55ac912df0b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912deb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1375341066 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911bfa60 0x55ac912de630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffba4c00640 0 client.4692 ms_handle_reset on v2:172.21.15.81:6837/328699254 2024-05-27T06:45:41.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 >> 172.21.15.81:0/1375341066 conn(0x55ac911e3db0 msgr2=0x55ac9131b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 shutdown_connections 2024-05-27T06:45:41.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1375341066 wait complete. 2024-05-27T06:45:41.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.325+0000 7ffbc5eb4000 20 client.4692 trim_cache size 0 max 16384 2024-05-27T06:45:41.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 Processor -- start 2024-05-27T06:45:41.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 -- start start 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8202200 con 0x55ac911bfa60 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb815bd10 con 0x55ac911d5f90 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.345+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55ac910c6220 con 0x55ac911dd1f0 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41622/0 (socket says 172.21.15.81:41622) 2024-05-27T06:45:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 -- 172.21.15.81:0/1079201937 learned_addr learned my addr 172.21.15.81:0/1079201937 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 -- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 msgr2=0x55ac9138c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 -- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 msgr2=0x55ac912e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 -- 172.21.15.81:0/1079201937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac910ca880 con 0x55ac911bfa60 2024-05-27T06:45:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffba8057690 tx=0x7ffba805e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2466126ec117c45 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffba7400640 1 -- 172.21.15.81:0/1079201937 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba807c0c0 con 0x55ac911bfa60 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffba7400640 1 -- 172.21.15.81:0/1079201937 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba80643e0 con 0x55ac911bfa60 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffba7400640 1 -- 172.21.15.81:0/1079201937 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba8001070 con 0x55ac911bfa60 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1079201937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 msgr2=0x55ac912e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffba8057690 tx=0x7ffba805e040 comp rx=0 tx=0).stop 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1079201937 shutdown_connections 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac912e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1079201937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac912e4510 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1079201937 >> 172.21.15.81:0/1079201937 conn(0x55ac911e3db0 msgr2=0x55ac912e5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1079201937 shutdown_connections 2024-05-27T06:45:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1079201937 wait complete. 2024-05-27T06:45:41.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 -- start start 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.349+0000 7ffbc5eb4000 1 Processor -- start 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac9138f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ac9112c5e0 con 0x55ac911bfa60 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ac910c17d0 con 0x55ac911d5f90 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x55ac91131900 con 0x55ac911dd1f0 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 10 client.0 _ll_register_callbacks cb 0x7ffba0009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41632/0 (socket says 172.21.15.81:41632) 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 -- 172.21.15.81:0/1641398485 learned_addr learned my addr 172.21.15.81:0/1641398485 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac9138f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7e00640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 -- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 msgr2=0x55ac9138fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 -- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 msgr2=0x55ac9138f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac9138f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 -- 172.21.15.81:0/1641398485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac910e0ea0 con 0x55ac911bfa60 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac9138f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb7400640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffba8057dd0 tx=0x7ffbb0004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64130192a780e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba808d340 con 0x55ac911bfa60 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffba80578a0 con 0x55ac911bfa60 2024-05-27T06:45:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffba80643e0 con 0x55ac911bfa60 2024-05-27T06:45:41.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ffba8060b50 con 0x55ac911bfa60 2024-05-27T06:45:41.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 10 client.4698 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:41.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55ac911ada20 con 0x55ac911bfa60 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba80643e0 con 0x55ac911bfa60 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ffba805a870 con 0x55ac911bfa60 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 10 client.4698 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 10 client.4698 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 20 client.4698 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dbf830 0x55ac91336440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 4 client.4698 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 --> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] -- command(tid 0: {"prefix": "status"}) -- 0x55ac910c8de0 con 0x55ac90dbf830 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dbf830 0x55ac91336440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbb6a00640 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dbf830 0x55ac91336440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34b2b5832e8b7bde server_cookie=e071860678698817 in_seq=0 out_seq=0 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 10 client.4698 ms_handle_connect on v2:172.21.15.81:6837/328699254 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 1 -- 172.21.15.81:0/1641398485 <== mds.1 v2:172.21.15.81:6837/328699254 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ac910c8de0 con 0x55ac90dbf830 2024-05-27T06:45:41.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 10 client.4698 handle_command_reply: tid=0 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 client.4698 shutdown 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 msgr2=0x55ac9138f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffba8057dd0 tx=0x7ffbb0004ac0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 shutdown_connections 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:6837/328699254,v1:172.21.15.81:6839/328699254] conn(0x55ac90dbf830 0x55ac91336440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x55ac911dd1f0 0x55ac9138fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ac911d5f90 0x55ac9138f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 --2- 172.21.15.81:0/1641398485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ac911bfa60 0x55ac9138f230 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffba4c00640 0 client.4698 ms_handle_reset on v2:172.21.15.81:6837/328699254 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.353+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 >> 172.21.15.81:0/1641398485 conn(0x55ac911e3db0 msgr2=0x55ac9131b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 shutdown_connections 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbc5eb4000 1 -- 172.21.15.81:0/1641398485 wait complete. 2024-05-27T06:45:41.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbc5eb4000 20 client.4698 trim_cache size 0 max 16384 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "cb957b26-d515-43f2-962a-6a97b6477eea", 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 1, 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4287, 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 29.687025501000001, 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-05-27T06:45:41.355 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 21, 2024-05-27T06:45:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 21, 2024-05-27T06:45:41.356 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 44.375643545999999 2024-05-27T06:45:41.356 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:41.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 msgr2=0x7ffb94071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 0x7ffb94071910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbb80cbaa0 tx=0x7ffba0008040 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 msgr2=0x7ffbb80cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffba800c430 tx=0x7ffba800c8f0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 shutdown_connections 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffbb8152a50 0x7ffbb8153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffb9406f400 0x7ffb94071910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffbb80c4600 0x7ffbb81556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 --2- 172.21.15.81:0/2783723724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbb80727a0 0x7ffbb80cbe80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 >> 172.21.15.81:0/2783723724 conn(0x7ffbb806cd90 msgr2=0x7ffbb81512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 shutdown_connections 2024-05-27T06:45:41.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.357+0000 7ffbbe800640 1 -- 172.21.15.81:0/2783723724 wait complete. 2024-05-27T06:45:41.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-05-27T06:45:41.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 Processor -- start 2024-05-27T06:45:41.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 -- start start 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff670154660 0x7ff67015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff6700c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff67015a620 con 0x7ff6700cff60 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff6700c81b0 con 0x7ff670154660 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff677c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6700c7bf0 con 0x7ff6700c96a0 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff675c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff6700cff60 0x7ff67015dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60476/0 (socket says 172.21.15.81:60476) 2024-05-27T06:45:41.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff675c00640 1 -- 172.21.15.81:0/4224138180 learned_addr learned my addr 172.21.15.81:0/4224138180 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff675200640 1 --2- 172.21.15.81:0/4224138180 >> v2:172.21.15.106:3300/0 conn(0x7ff670154660 0x7ff67015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff667e00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff6700c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889133382 0 0) 0x7ff67015a620 con 0x7ff6700cff60 2024-05-27T06:45:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.525+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff654003530 con 0x7ff6700cff60 2024-05-27T06:45:41.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4035090648 0 0) 0x7ff654003530 con 0x7ff6700cff60 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 msgr2=0x7ff6700c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff6700c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 >> v2:172.21.15.106:3300/0 conn(0x7ff670154660 msgr2=0x7ff67015bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 --2- 172.21.15.81:0/4224138180 >> v2:172.21.15.106:3300/0 conn(0x7ff670154660 0x7ff67015bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6700c7ed0 con 0x7ff6700cff60 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7ff668001a80 con 0x7ff6700cff60 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 >> v1:172.21.15.81:6789/0 conn(0x7ff6700cff60 legacy=0x7ff67015dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff654003a10 0x7ff654003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff654004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff654004d50 0x7ff654005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6700c7bf0 con 0x7ff654003a10 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff654009ac0 con 0x7ff654004d50 2024-05-27T06:45:41.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6540145d0 con 0x7ff6700c96a0 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff654003a10 0x7ff654003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675c00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff654004d50 0x7ff654005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff667e00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff654004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 -- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 msgr2=0x7ff654004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff654004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 -- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff654004d50 msgr2=0x7ff654005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff654004d50 0x7ff654005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff675200640 1 -- 172.21.15.81:0/4224138180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff654003530 con 0x7ff654003a10 2024-05-27T06:45:41.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.529+0000 7ff667e00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6700c96a0 0x7ff654004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff675200640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff654003a10 0x7ff654003e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff660009ab0 tx=0x7ff66002da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879078961d79f3f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600c3070 con 0x7ff654003a10 2024-05-27T06:45:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6600b6760 con 0x7ff654003a10 2024-05-27T06:45:41.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff66fe00640 1 -- 172.21.15.81:0/4224138180 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600096b0 con 0x7ff654003a10 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- 172.21.15.81:0/4224138180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff654003a10 msgr2=0x7ff654003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 --2- 172.21.15.81:0/4224138180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff654003a10 0x7ff654003e80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff660009ab0 tx=0x7ff66002da10 comp rx=0 tx=0).stop 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff675c00640 1 -- 172.21.15.81:0/4224138180 reap_dead start 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- 172.21.15.81:0/4224138180 shutdown_connections 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- 172.21.15.81:0/4224138180 >> 172.21.15.81:0/4224138180 conn(0x7ff670068ab0 msgr2=0x7ff670068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- 172.21.15.81:0/4224138180 shutdown_connections 2024-05-27T06:45:41.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- 172.21.15.81:0/4224138180 wait complete. 2024-05-27T06:45:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 Processor -- start 2024-05-27T06:45:41.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.533+0000 7ff677c00640 1 -- start start 2024-05-27T06:45:41.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff654003a10 0x7ff6701f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6701595c0 0x7ff67014f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff670154a70 con 0x7ff670150820 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff6700c9210 con 0x7ff6701595c0 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6700d0780 con 0x7ff654003a10 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41662/0 (socket says 172.21.15.81:41662) 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 -- 172.21.15.81:0/3162856817 learned_addr learned my addr 172.21.15.81:0/3162856817 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff675c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6701595c0 0x7ff67014f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff675200640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff654003a10 0x7ff6701f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 -- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff654003a10 msgr2=0x7ff6701f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff654003a10 0x7ff6701f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 -- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6701595c0 msgr2=0x7ff67014f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6701595c0 0x7ff67014f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 -- 172.21.15.81:0/3162856817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6700cc110 con 0x7ff670150820 2024-05-27T06:45:41.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff667e00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6580116a0 tx=0x7ff658011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff66d600640 1 -- 172.21.15.81:0/3162856817 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658004220 con 0x7ff670150820 2024-05-27T06:45:41.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff66d600640 1 -- 172.21.15.81:0/3162856817 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6580043c0 con 0x7ff670150820 2024-05-27T06:45:41.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff66d600640 1 -- 172.21.15.81:0/3162856817 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658015870 con 0x7ff670150820 2024-05-27T06:45:41.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67015a630 con 0x7ff670150820 2024-05-27T06:45:41.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.537+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6700cb0f0 con 0x7ff670150820 2024-05-27T06:45:41.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff66d600640 1 -- 172.21.15.81:0/3162856817 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff658015a10 con 0x7ff670150820 2024-05-27T06:45:41.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff66d600640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 0x7ff64c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff66d600640 1 -- 172.21.15.81:0/3162856817 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff65808f800 con 0x7ff670150820 2024-05-27T06:45:41.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff67f3a5000 1 Processor -- start 2024-05-27T06:45:41.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff675200640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 0x7ff64c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff67f3a5000 1 -- start start 2024-05-27T06:45:41.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.541+0000 7ff675200640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 0x7ff64c071910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c0000 tx=0x7ff6600046d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a4318fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 0x5589a4319500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5589a416d6f0 con 0x5589a41f3a60 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5589a415d290 con 0x5589a42111f0 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x5589a41612d0 con 0x5589a4209f90 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41676/0 (socket says 172.21.15.81:41676) 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 -- 172.21.15.81:0/1801671333 learned_addr learned my addr 172.21.15.81:0/1801671333 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff667e00640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 0x5589a4319500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675200640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a4318fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 -- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 msgr2=0x5589a4319500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 0x5589a4319500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 -- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 msgr2=0x5589a4318fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a4318fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 -- 172.21.15.81:0/1801671333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589a413bc10 con 0x5589a41f3a60 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff667e00640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 0x5589a4319500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff675c00640 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff670071ca0 tx=0x7ff66800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603e21d59d47b32e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff665600640 1 -- 172.21.15.81:0/1801671333 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668007980 con 0x5589a41f3a60 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff665600640 1 -- 172.21.15.81:0/1801671333 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66801a020 con 0x5589a41f3a60 2024-05-27T06:45:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff665600640 1 -- 172.21.15.81:0/1801671333 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668015440 con 0x5589a41f3a60 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- 172.21.15.81:0/1801671333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 msgr2=0x5589a4351bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff670071ca0 tx=0x7ff66800a060 comp rx=0 tx=0).stop 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- 172.21.15.81:0/1801671333 shutdown_connections 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a4209f90 0x5589a4319500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a41f3a60 0x5589a4351bd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/1801671333 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a4318fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- 172.21.15.81:0/1801671333 >> 172.21.15.81:0/1801671333 conn(0x5589a4217db0 msgr2=0x5589a434f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- 172.21.15.81:0/1801671333 shutdown_connections 2024-05-27T06:45:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- 172.21.15.81:0/1801671333 wait complete. 2024-05-27T06:45:41.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 -- start start 2024-05-27T06:45:41.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.545+0000 7ff67f3a5000 1 Processor -- start 2024-05-27T06:45:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a4312660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43130e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5589a4120650 con 0x5589a4209f90 2024-05-27T06:45:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5589a41e1590 con 0x5589a41f3a60 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x5589a416c4c0 con 0x5589a42111f0 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 10 client.0 _ll_register_callbacks cb 0x5589a42fc560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41684/0 (socket says 172.21.15.81:41684) 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 -- 172.21.15.81:0/3853023834 learned_addr learned my addr 172.21.15.81:0/3853023834 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff675c00640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff675200640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a4312660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 -- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 msgr2=0x5589a43130e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43130e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 -- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 msgr2=0x5589a4312660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a4312660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 -- 172.21.15.81:0/3853023834 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589a41a59d0 con 0x5589a4209f90 2024-05-27T06:45:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff675200640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a4312660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff667e00640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff658006c10 tx=0x7ff65805c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9f4a360906ddaf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658022070 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff658057cb0 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff66801a020 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff65805bc50 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 10 client.4710 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5589a4138530 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff66801a020 con 0x5589a4209f90 2024-05-27T06:45:41.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff65805ae10 con 0x5589a4209f90 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 10 client.4710 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 10 client.4710 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 20 client.4710 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3e057d0 0x5589a4313af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 4 client.4710 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 --> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589a40f8be0 con 0x5589a3e057d0 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.549+0000 7ff675200640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3e057d0 0x5589a4313af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff675200640 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3e057d0 0x5589a4313af0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7bede0228c9ebd5 server_cookie=b0506e849fd081d8 in_seq=0 out_seq=0 2024-05-27T06:45:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff65e000640 10 client.4710 ms_handle_connect on v2:172.21.15.106:6834/730205609 2024-05-27T06:45:41.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff65e000640 1 -- 172.21.15.81:0/3853023834 <== mds.0 v2:172.21.15.106:6834/730205609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24041 (crc 0 0 0) 0x5589a40f8be0 con 0x5589a3e057d0 2024-05-27T06:45:41.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff65e000640 10 client.4710 handle_command_reply: tid=0 2024-05-27T06:45:41.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 client.4710 shutdown 2024-05-27T06:45:41.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 msgr2=0x5589a4312ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff658006c10 tx=0x7ff65805c740 comp rx=0 tx=0).stop 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 shutdown_connections 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3e057d0 0x5589a4313af0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43130e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a4312ba0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/3853023834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a4312660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff65e000640 0 client.4710 ms_handle_reset on v2:172.21.15.106:6834/730205609 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 >> 172.21.15.81:0/3853023834 conn(0x5589a4217db0 msgr2=0x5589a434f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 shutdown_connections 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 1 -- 172.21.15.81:0/3853023834 wait complete. 2024-05-27T06:45:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.553+0000 7ff67f3a5000 20 client.4710 trim_cache size 0 max 16384 2024-05-27T06:45:41.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 Processor -- start 2024-05-27T06:45:41.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 -- start start 2024-05-27T06:45:41.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a436bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43bf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff67006e9c0 con 0x5589a4209f90 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5589a40fa220 con 0x5589a41f3a60 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x5589a40f57d0 con 0x5589a42111f0 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41688/0 (socket says 172.21.15.81:41688) 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 -- 172.21.15.81:0/960888034 learned_addr learned my addr 172.21.15.81:0/960888034 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff675200640 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a436bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff675c00640 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43bf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 -- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 msgr2=0x5589a43bf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43bf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 -- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 msgr2=0x5589a436bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a436bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 -- 172.21.15.81:0/960888034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589a40fe880 con 0x5589a4209f90 2024-05-27T06:45:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff667e00640 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff6580593b0 tx=0x7ff65805e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88429dd57a083787 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff664c00640 1 -- 172.21.15.81:0/960888034 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658022070 con 0x5589a4209f90 2024-05-27T06:45:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff664c00640 1 -- 172.21.15.81:0/960888034 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65805c140 con 0x5589a4209f90 2024-05-27T06:45:41.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.573+0000 7ff664c00640 1 -- 172.21.15.81:0/960888034 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658060440 con 0x5589a4209f90 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- 172.21.15.81:0/960888034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 msgr2=0x5589a436c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff6580593b0 tx=0x7ff65805e000 comp rx=0 tx=0).stop 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- 172.21.15.81:0/960888034 shutdown_connections 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a42111f0 0x5589a43bf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a436c530 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/960888034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a41f3a60 0x5589a436bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- 172.21.15.81:0/960888034 >> 172.21.15.81:0/960888034 conn(0x5589a4217db0 msgr2=0x5589a4318bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- 172.21.15.81:0/960888034 shutdown_connections 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- 172.21.15.81:0/960888034 wait complete. 2024-05-27T06:45:41.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- start start 2024-05-27T06:45:41.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 Processor -- start 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a41f3a60 0x5589a4389ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a438a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5589a4165900 con 0x5589a4209f90 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5589a41605e0 con 0x5589a42111f0 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x5589a4120650 con 0x5589a41f3a60 2024-05-27T06:45:41.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff67f3a5000 10 client.0 _ll_register_callbacks cb 0x7ff658004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41700/0 (socket says 172.21.15.81:41700) 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 -- 172.21.15.81:0/2078482474 learned_addr learned my addr 172.21.15.81:0/2078482474 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff675200640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a41f3a60 0x5589a4389ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff675c00640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a438a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 -- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a41f3a60 msgr2=0x5589a4389ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a41f3a60 0x5589a4389ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 -- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 msgr2=0x5589a438a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a438a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 -- 172.21.15.81:0/2078482474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589a4114ea0 con 0x5589a4209f90 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff667e00640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff6580586e0 tx=0x7ff65808dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4819f3c61f4019ca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.577+0000 7ff65e000640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-05-27T06:45:41.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658022040 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff65805fb10 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff65805c140 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7ff65805bf10 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 10 client.4716 fetch_fsmap learned FSMap version 17 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5589a41e1a20 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65808c080 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7ff65805f770 con 0x5589a4209f90 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 10 client.4716 fetch_fsmap finished waiting for FSMap version 17 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 10 client.4716 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 20 client.4716 populate_metadata read hostname 'smithi081' 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3df3830 0x5589a43c9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 4 client.4716 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-27T06:45:41.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 --> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] -- command(tid 0: {"prefix": "status"}) -- 0x5589a40fcde0 con 0x5589a3df3830 2024-05-27T06:45:41.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff675200640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3df3830 0x5589a43c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 10 client.4716 ms_handle_connect on v2:172.21.15.106:6834/730205609 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff675200640 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3df3830 0x5589a43c9520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6800115398e46f8f server_cookie=4feb6b1811db9583 in_seq=0 out_seq=0 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 1 -- 172.21.15.81:0/2078482474 <== mds.0 v2:172.21.15.106:6834/730205609 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5589a40fcde0 con 0x5589a3df3830 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 10 client.4716 handle_command_reply: tid=0 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 client.4716 shutdown 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 msgr2=0x5589a438a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff6580586e0 tx=0x7ff65808dc90 comp rx=0 tx=0).stop 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 shutdown_connections 2024-05-27T06:45:41.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:6834/730205609,v1:172.21.15.106:6836/730205609] conn(0x5589a3df3830 0x5589a43c9520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5589a42111f0 0x5589a438a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589a4209f90 0x5589a438a220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 --2- 172.21.15.81:0/2078482474 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x5589a41f3a60 0x5589a4389ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff65e000640 0 client.4716 ms_handle_reset on v2:172.21.15.106:6834/730205609 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 >> 172.21.15.81:0/2078482474 conn(0x5589a4217db0 msgr2=0x5589a43191e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 shutdown_connections 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 1 -- 172.21.15.81:0/2078482474 wait complete. 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff67f3a5000 20 client.4716 trim_cache size 0 max 16384 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "cb957b26-d515-43f2-962a-6a97b6477eea", 2024-05-27T06:45:41.581 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4302, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 38.958175224000001, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 21, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 21, 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 44.586159801000001 2024-05-27T06:45:41.582 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 msgr2=0x7ff64c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 0x7ff64c071910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff6600c0000 tx=0x7ff6600046d0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 msgr2=0x7ff6700cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6580116a0 tx=0x7ff658011aa0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 shutdown_connections 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff64c06f400 0x7ff64c071910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6701595c0 0x7ff67014f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670150820 0x7ff6700cbbd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 --2- 172.21.15.81:0/3162856817 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff654003a10 0x7ff6701f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 >> 172.21.15.81:0/3162856817 conn(0x7ff670068ab0 msgr2=0x7ff67015b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 shutdown_connections 2024-05-27T06:45:41.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.581+0000 7ff677c00640 1 -- 172.21.15.81:0/3162856817 wait complete. 2024-05-27T06:45:41.598 INFO:teuthology.run_tasks:Running task sequential... 2024-05-27T06:45:41.610 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-27T06:45:41.610 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-27T06:45:41.611 INFO:teuthology.task.print:Enabling mgr modules 2024-05-27T06:45:41.611 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-27T06:45:41.719 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-27T06:45:41.719 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-05-27T06:45:41.720 INFO:tasks.ceph_fuse:client.0 config is {} 2024-05-27T06:45:41.720 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-27T06:45:41.720 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:41.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:41.720 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-05-27T06:45:41.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:41.730 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-05-27T06:45:41.792 INFO:teuthology.orchestra.run.smithi081.stderr:Cannot find device "ceph-brx" 2024-05-27T06:45:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T06:45:41.795 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-27T06:45:41.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-27T06:45:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.993+0000 7fcc01e00640 1 Processor -- start 2024-05-27T06:45:41.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.993+0000 7fcc01e00640 1 -- start start 2024-05-27T06:45:41.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcc01e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcbfc0ce220 0x7fcbfc153960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcc01e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc06ec30 0x7fcbfc06f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcc01e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc0c8160 con 0x7fcbfc1598f0 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcc01e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fcbfc0c75e0 con 0x7fcbfc0ce220 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcc01e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0c78c0 con 0x7fcbfc06ec30 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcbfc0ce220 0x7fcbfc153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfb400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcbfc1598f0 0x7fcbfc159d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60480/0 (socket says 172.21.15.81:60480) 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfb400640 1 -- 172.21.15.81:0/116790936 learned_addr learned my addr 172.21.15.81:0/116790936 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcbfc0ce220 0x7fcbfc153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41716/0 (socket says 172.21.15.81:41716) 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbf3e00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc06ec30 0x7fcbfc06f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 -- 172.21.15.81:0/116790936 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc06ec30 msgr2=0x7fcbfc06f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc06ec30 0x7fcbfc06f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 -- 172.21.15.81:0/116790936 >> v1:172.21.15.81:6789/0 conn(0x7fcbfc1598f0 legacy=0x7fcbfc159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:41.997+0000 7fcbfaa00640 1 -- 172.21.15.81:0/116790936 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc0c7ba0 con 0x7fcbfc0ce220 2024-05-27T06:45:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcbfaa00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc153960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0097c0 tx=0x7fcbec01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f971cc15567b344 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcbfa000640 1 -- 172.21.15.81:0/116790936 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b0070 con 0x7fcbfc0ce220 2024-05-27T06:45:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcbfa000640 1 -- 172.21.15.81:0/116790936 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0ab600 con 0x7fcbfc0ce220 2024-05-27T06:45:41.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcbfa000640 1 -- 172.21.15.81:0/116790936 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0ab9d0 con 0x7fcbfc0ce220 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- 172.21.15.81:0/116790936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc0ce220 msgr2=0x7fcbfc153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc153960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0097c0 tx=0x7fcbec01c3c0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- 172.21.15.81:0/116790936 shutdown_connections 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc06ec30 0x7fcbfc06f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 --2- 172.21.15.81:0/116790936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc153960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- 172.21.15.81:0/116790936 >> 172.21.15.81:0/116790936 conn(0x7fcbfc0cb2e0 msgr2=0x7fcbfc0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- 172.21.15.81:0/116790936 shutdown_connections 2024-05-27T06:45:41.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- 172.21.15.81:0/116790936 wait complete. 2024-05-27T06:45:41.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 Processor -- start 2024-05-27T06:45:41.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.001+0000 7fcc01e00640 1 -- start start 2024-05-27T06:45:42.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:42.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 0x7fcbfc1411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0c57c0 con 0x7fcbfc06ec30 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0c5d30 con 0x7fcbfc0ce220 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0c5940 con 0x7fcbfc1598f0 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41724/0 (socket says 172.21.15.81:41724) 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 -- 172.21.15.81:0/3291225546 learned_addr learned my addr 172.21.15.81:0/3291225546 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfb400640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 0x7fcbfc1411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbf3e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 -- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 msgr2=0x7fcbfc1411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 0x7fcbfc1411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 -- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbfc0ce220 msgr2=0x7fcbfc144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 -- 172.21.15.81:0/3291225546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc145b00 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfb400640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 0x7fcbfc1411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbfaa00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcbec01aa70 tx=0x7fcbec01ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b0050 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0b1040 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc1f4b00 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0aa9e0 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1f50c0 con 0x7fcbfc06ec30 2024-05-27T06:45:42.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.005+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc06a560 con 0x7fcbfc06ec30 2024-05-27T06:45:42.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.009+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcbec0be030 con 0x7fcbfc06ec30 2024-05-27T06:45:42.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.009+0000 7fcbf3400640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 0x7fcbd8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:42.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.009+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcbec129220 con 0x7fcbfc06ec30 2024-05-27T06:45:42.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.009+0000 7fcbf3e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 0x7fcbd8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:42.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.009+0000 7fcbf3e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 0x7fcbd8071940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4006fd0 tx=0x7fcbe4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:42.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.013+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fcbec015fb0 con 0x7fcbfc06ec30 2024-05-27T06:45:42.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.205+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcbfc0cdbe0 con 0x7fcbfc06ec30 2024-05-27T06:45:42.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.205+0000 7fcbf3400640 1 -- 172.21.15.81:0/3291225546 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fcbec0f6190 con 0x7fcbfc06ec30 2024-05-27T06:45:42.203 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 msgr2=0x7fcbd8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 0x7fcbd8071940 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4006fd0 tx=0x7fcbe4008040 comp rx=0 tx=0).stop 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 msgr2=0x7fcbfc1455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcbec01aa70 tx=0x7fcbec01ac00 comp rx=0 tx=0).stop 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 shutdown_connections 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcbfc1598f0 0x7fcbfc1411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcbd806f430 0x7fcbd8071940 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbfc0ce220 0x7fcbfc144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 --2- 172.21.15.81:0/3291225546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbfc06ec30 0x7fcbfc1455c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:42.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 >> 172.21.15.81:0/3291225546 conn(0x7fcbfc0cb2e0 msgr2=0x7fcbfc070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:42.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 shutdown_connections 2024-05-27T06:45:42.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42.209+0000 7fcc01e00640 1 -- 172.21.15.81:0/3291225546 wait complete. 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-27T06:45:42.225 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-27T06:45:42.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:42.226 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-05-27T06:45:42.235 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42462sec preferred_lft 42462sec 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-05-27T06:45:42.236 INFO:teuthology.orchestra.run.smithi081.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-05-27T06:45:42.237 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 02:42:c3:18:e3:3c brd ff:ff:ff:ff:ff:ff 2024-05-27T06:45:42.237 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-05-27T06:45:42.237 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-05-27T06:45:42.237 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-27T06:45:42.237 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T06:45:42.237 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-05-27T06:45:42.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2024-05-27T06:45:42.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2024-05-27T06:45:42.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2024-05-27T06:45:42.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-27T06:45:42.238 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-05-27T06:45:42.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T06:45:42.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:42.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:42.391 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-27T06:45:42.456 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-05-27T06:45:42.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:42.459 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-05-27T06:45:42.535 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-27T06:45:42.536 INFO:teuthology.orchestra.run.smithi081.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T06:45:42.536 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T06:45:42.536 INFO:teuthology.orchestra.run.smithi081.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-05-27T06:45:42.536 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-27T06:45:42.536 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-05-27T06:45:42.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T06:45:42.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:42.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:42.680 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-05-27T06:45:42.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:42.689 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-05-27T06:45:42.739 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T06:45:42.739 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-05-27T06:45:42.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:42.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-27T06:45:42.739 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-05-27T06:45:42.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T06:45:42.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:42.870 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-27T06:45:42.870 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-27T06:45:42.871 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-05-27T06:45:42.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T06:45:43.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:43.034 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-27T06:45:43.034 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-05-27T06:45:43.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-05-27T06:45:43.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2024-05-27T06:45:43.034 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-05-27T06:45:43.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-27T06:45:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:43.159 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-27T06:45:43.159 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T06:45:43.159 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:43.215 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-27T06:45:43.216 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-27T06:45:43.216 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:43.266 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2024-05-27T06:45:43.327 DEBUG:teuthology.orchestra.run.smithi081:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T06:45:43.384 INFO:teuthology.orchestra.run.smithi081.stdout:/sys 2024-05-27T06:45:43.384 INFO:teuthology.orchestra.run.smithi081.stdout:/proc 2024-05-27T06:45:43.384 INFO:teuthology.orchestra.run.smithi081.stdout:/dev 2024-05-27T06:45:43.384 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/pts 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/run 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/ 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/security 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/shm 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/run/lock 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/cgroup 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/pstore 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/bpf 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/hugepages 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/mqueue 2024-05-27T06:45:43.385 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/debug 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/tracing 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/fs/nfsd 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/fuse/connections 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/config 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-sysusers.service 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/run/rpc_pipefs 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/1000 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-0 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-1 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-2 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-3 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns 2024-05-27T06:45:43.386 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:43.387 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:43.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:43.387 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2024-05-27T06:45:43.438 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-27T06:45:43.438 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-27T06:45:43.485 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2024-05-27T06:45:43.508 DEBUG:teuthology.orchestra.run.smithi081:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T06:45:43.560 INFO:teuthology.orchestra.run.smithi081.stdout:/sys 2024-05-27T06:45:43.560 INFO:teuthology.orchestra.run.smithi081.stdout:/proc 2024-05-27T06:45:43.560 INFO:teuthology.orchestra.run.smithi081.stdout:/dev 2024-05-27T06:45:43.560 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/pts 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/run 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/ 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/security 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/shm 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/run/lock 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/cgroup 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/pstore 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/bpf 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/hugepages 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/mqueue 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/debug 2024-05-27T06:45:43.561 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/tracing 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/fs/nfsd 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/fuse/connections 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/config 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-sysusers.service 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/rpc_pipefs 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/1000 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-0 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-1 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-2 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-3 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:43.562 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:43.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:43.563 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2024-05-27T06:45:43.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi081.stderr:2024-05-27T06:45:43.601+0000 7fdb2e067580 -1 init, newargv = 0x7fdb14001280 newargc=15 2024-05-27T06:45:43.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi081.stderr:ceph-fuse[33847]: starting ceph client 2024-05-27T06:45:43.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi081.stderr:ceph-fuse[33847]: starting fuse 2024-05-27T06:45:44.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2024-05-27T06:45:44.734 DEBUG:teuthology.orchestra.run.smithi081:> cat /proc/self/mounts | awk '{print $2}' 2024-05-27T06:45:44.789 INFO:teuthology.orchestra.run.smithi081.stdout:/sys 2024-05-27T06:45:44.789 INFO:teuthology.orchestra.run.smithi081.stdout:/proc 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/dev 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/pts 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/run 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/ 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/security 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/shm 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/run/lock 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/cgroup 2024-05-27T06:45:44.790 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/pstore 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/bpf 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/hugepages 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/mqueue 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/debug 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/tracing 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/fs/nfsd 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/fuse/connections 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/config 2024-05-27T06:45:44.791 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-sysusers.service 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/run/rpc_pipefs 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/1000 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-0 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-1 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-2 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph/osd/ceph-3 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T06:45:44.792 INFO:teuthology.orchestra.run.smithi081.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:44.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:44.793 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2024-05-27T06:45:44.847 INFO:teuthology.orchestra.run.smithi081.stdout:44 2024-05-27T06:45:44.847 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [44] 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo stdin-killer -- python3 -c ' 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> import glob 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> import re 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> import os 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> import subprocess 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> def _find_admin_socket(client_name): 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> files = glob.glob(asok_path) 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> # Given a non-glob path, it better be there 2024-05-27T06:45:44.848 DEBUG:teuthology.orchestra.run.smithi081:> if "*" not in asok_path: 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> assert(len(files) == 1) 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> return files[0] 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> for f in files: 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> contents = proc_f.read() 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> if mountpoint in contents: 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> return f 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> print(_find_admin_socket("client.0")) 2024-05-27T06:45:44.849 DEBUG:teuthology.orchestra.run.smithi081:> ' 2024-05-27T06:45:44.981 INFO:teuthology.orchestra.run.smithi081.stdout:/var/run/ceph/ceph-client.0.33847.asok 2024-05-27T06:45:44.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-27T06:45:44.993 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.33847.asok 2024-05-27T06:45:44.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:44.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.33847.asok status 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "metadata": { 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_sha1": "38f9d900c83c63799fbdbe61acc9a11b0d3554a6", 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev)", 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "entity_id": "0", 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "hostname": "smithi081", 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-27T06:45:45.173 INFO:teuthology.orchestra.run.smithi081.stdout: "pid": "33847", 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "root": "/" 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_count": 0, 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_pinned_count": 0, 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 4702, 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "inst": { 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "name": { 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client", 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "num": 4702 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-05-27T06:45:45.174 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:0", 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3289699705 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "172.21.15.81:0", 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 3289699705 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "inst_str": "client.4702 172.21.15.81:0/3289699705", 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "addr_str": "172.21.15.81:0/3289699705", 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "inode_count": 1, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_epoch": 17, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch": 22, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch_barrier": 0, 2024-05-27T06:45:45.175 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklisted": false, 2024-05-27T06:45:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs" 2024-05-27T06:45:45.176 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-05-27T06:45:45.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:45.188 DEBUG:teuthology.orchestra.run.smithi081:> stat --file-system '--printf=%T 2024-05-27T06:45:45.188 DEBUG:teuthology.orchestra.run.smithi081:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:45.198 INFO:teuthology.orchestra.run.smithi081.stdout:fuseblk 2024-05-27T06:45:45.198 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:45.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T06:45:45.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:45.264 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-05-27T06:45:45.300 INFO:tasks.mds_thrash:mds thrasher using random seed: 1716792345 2024-05-27T06:45:45.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:45.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 Processor -- start 2024-05-27T06:45:45.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 -- start start 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f7690165d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76900c78b0 con 0x7f76900ca770 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76900c6ee0 con 0x7f76900cfdb0 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7696e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f76900c71c0 con 0x7f76900d01c0 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768fe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f76900ca770 0x7f76900cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34242/0 (socket says 172.21.15.81:34242) 2024-05-27T06:45:45.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768fe00640 1 -- 172.21.15.81:0/1493774458 learned_addr learned my addr 172.21.15.81:0/1493774458 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:45.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7694e00640 1 --2- 172.21.15.81:0/1493774458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f7690165d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1154253914 0 0) 0x7f76900c78b0 con 0x7f76900ca770 2024-05-27T06:45:45.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7674003530 con 0x7f76900ca770 2024-05-27T06:45:45.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4016725004 0 0) 0x7f76900c6ee0 con 0x7f76900cfdb0 2024-05-27T06:45:45.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76900c78b0 con 0x7f76900cfdb0 2024-05-27T06:45:45.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1973860184 0 0) 0x7f7674003530 con 0x7f76900ca770 2024-05-27T06:45:45.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7694e00640 1 -- 172.21.15.81:0/1493774458 >> v1:172.21.15.106:6789/0 conn(0x7f76900cfdb0 legacy=0x7f7690154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.457+0000 7f7694e00640 1 -- 172.21.15.81:0/1493774458 >> v1:172.21.15.81:6789/0 conn(0x7f76900ca770 legacy=0x7f76900cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.461+0000 7f7694e00640 1 -- 172.21.15.81:0/1493774458 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76900c7b70 con 0x7f76900d01c0 2024-05-27T06:45:45.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.461+0000 7f7694e00640 1 --2- 172.21.15.81:0/1493774458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f7690165d80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7688009dc0 tx=0x7f768801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebaab62b7a94aef2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.461+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76880b2070 con 0x7f76900d01c0 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.461+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76880a5600 con 0x7f76900d01c0 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.461+0000 7f768ea00640 1 -- 172.21.15.81:0/1493774458 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680001700 con 0x7f76900d01c0 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- 172.21.15.81:0/1493774458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 msgr2=0x7f7690165d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 --2- 172.21.15.81:0/1493774458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f7690165d80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7688009dc0 tx=0x7f768801c8b0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- 172.21.15.81:0/1493774458 shutdown_connections 2024-05-27T06:45:45.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 --2- 172.21.15.81:0/1493774458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f7690165d80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- 172.21.15.81:0/1493774458 >> 172.21.15.81:0/1493774458 conn(0x7f7690068ae0 msgr2=0x7f7690068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:45.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- 172.21.15.81:0/1493774458 shutdown_connections 2024-05-27T06:45:45.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- 172.21.15.81:0/1493774458 wait complete. 2024-05-27T06:45:45.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 Processor -- start 2024-05-27T06:45:45.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.465+0000 7f7696e00640 1 -- start start 2024-05-27T06:45:45.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76900cfdb0 0x7f769014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f769006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76900cc050 con 0x7f76900ca770 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f76900cbed0 con 0x7f76900cfdb0 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7690154910 con 0x7f76900d01c0 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44110/0 (socket says 172.21.15.81:44110) 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 -- 172.21.15.81:0/2662849503 learned_addr learned my addr 172.21.15.81:0/2662849503 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768f400640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76900cfdb0 0x7f769014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7694e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f769006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 -- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 msgr2=0x7f769006e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f769006e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 -- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76900cfdb0 msgr2=0x7f769014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76900cfdb0 0x7f769014bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 -- 172.21.15.81:0/2662849503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7690159140 con 0x7f76900ca770 2024-05-27T06:45:45.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f768fe00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f768000c6e0 tx=0x7f768000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680005ae0 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7680005c80 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76901f5bc0 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680007290 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7690158120 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f769006a560 con 0x7f76900ca770 2024-05-27T06:45:45.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.469+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7680007430 con 0x7f76900ca770 2024-05-27T06:45:45.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.473+0000 7f7687e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 0x7f7670071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.473+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f768008f070 con 0x7f76900ca770 2024-05-27T06:45:45.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.473+0000 7f768f400640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 0x7f7670071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.473+0000 7f768f400640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 0x7f7670071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7690153eb0 tx=0x7f767c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.477+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7680007720 con 0x7f76900ca770 2024-05-27T06:45:45.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.637+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76900c6610 con 0x7f76900ca770 2024-05-27T06:45:45.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.637+0000 7f7687e00640 1 -- 172.21.15.81:0/2662849503 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16781 (secure 0 0 0) 0x7f768005bed0 con 0x7f76900ca770 2024-05-27T06:45:45.634 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:45.634 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:44.606834+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 msgr2=0x7f7670071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 0x7f7670071910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7690153eb0 tx=0x7f767c009040 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 msgr2=0x7f7690158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f768000c6e0 tx=0x7f768000cba0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 shutdown_connections 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f76900d01c0 0x7f769006e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f767006f400 0x7f7670071910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76900cfdb0 0x7f769014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 --2- 172.21.15.81:0/2662849503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76900ca770 0x7f7690158c00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 >> 172.21.15.81:0/2662849503 conn(0x7f7690068ae0 msgr2=0x7f76900cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 shutdown_connections 2024-05-27T06:45:45.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.641+0000 7f7696e00640 1 -- 172.21.15.81:0/2662849503 wait complete. 2024-05-27T06:45:45.656 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-27T06:44:51.516548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-27T06:45:00.358885+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-27T06:45:00.967125+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-27T06:45:45.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-27T06:45:45.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.809+0000 7fe98aa00640 1 Processor -- start 2024-05-27T06:45:45.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.809+0000 7fe98aa00640 1 -- start start 2024-05-27T06:45:45.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe98415fdc0 0x7fe98415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 0x7fe9840ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe98406e120 0x7fe98406e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe98406ea70 con 0x7fe98415fdc0 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe9841603e0 con 0x7fe9840ca2e0 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe9841606c0 con 0x7fe98406e120 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe98415fdc0 0x7fe98415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe98415fdc0 0x7fe98415d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44126/0 (socket says 172.21.15.81:44126) 2024-05-27T06:45:45.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 -- 172.21.15.81:0/3829367685 learned_addr learned my addr 172.21.15.81:0/3829367685 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe98a000640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe98406e120 0x7fe98406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe988c00640 1 --2- 172.21.15.81:0/3829367685 >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 0x7fe9840ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 -- 172.21.15.81:0/3829367685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe98406e120 msgr2=0x7fe98406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe98406e120 0x7fe98406e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 -- 172.21.15.81:0/3829367685 >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 msgr2=0x7fe9840ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 --2- 172.21.15.81:0/3829367685 >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 0x7fe9840ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe989600640 1 -- 172.21.15.81:0/3829367685 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9841609a0 con 0x7fe98415fdc0 2024-05-27T06:45:45.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.813+0000 7fe988c00640 1 --2- 172.21.15.81:0/3829367685 >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 0x7fe9840ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:45.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe989600640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98415fdc0 0x7fe98415d0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe984072f20 tx=0x7fe97801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc38f3b60d306976 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe983e00640 1 -- 172.21.15.81:0/3829367685 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780b0070 con 0x7fe98415fdc0 2024-05-27T06:45:45.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe983e00640 1 -- 172.21.15.81:0/3829367685 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9780ab600 con 0x7fe98415fdc0 2024-05-27T06:45:45.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe983e00640 1 -- 172.21.15.81:0/3829367685 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780ab9d0 con 0x7fe98415fdc0 2024-05-27T06:45:45.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 -- 172.21.15.81:0/3829367685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98415fdc0 msgr2=0x7fe98415d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98415fdc0 0x7fe98415d0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe984072f20 tx=0x7fe97801c4f0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 -- 172.21.15.81:0/3829367685 shutdown_connections 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe98406e120 0x7fe98406e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3829367685 >> v2:172.21.15.106:3300/0 conn(0x7fe9840ca2e0 0x7fe9840ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3829367685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98415fdc0 0x7fe98415d0e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 -- 172.21.15.81:0/3829367685 >> 172.21.15.81:0/3829367685 conn(0x7fe984068cb0 msgr2=0x7fe9840720f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 -- 172.21.15.81:0/3829367685 shutdown_connections 2024-05-27T06:45:45.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.817+0000 7fe98aa00640 1 -- 172.21.15.81:0/3829367685 wait complete. 2024-05-27T06:45:45.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 Processor -- start 2024-05-27T06:45:45.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- start start 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9840ca2e0 0x7fe9840cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe98415fdc0 0x7fe984159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe98415d7a0 con 0x7fe98406e120 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe98415d620 con 0x7fe98415fdc0 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe984074d40 con 0x7fe9840ca2e0 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44136/0 (socket says 172.21.15.81:44136) 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 -- 172.21.15.81:0/3958914408 learned_addr learned my addr 172.21.15.81:0/3958914408 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98a000640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe98415fdc0 0x7fe984159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe988c00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9840ca2e0 0x7fe9840cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 -- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9840ca2e0 msgr2=0x7fe9840cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9840ca2e0 0x7fe9840cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 -- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe98415fdc0 msgr2=0x7fe984159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe98415fdc0 0x7fe984159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 -- 172.21.15.81:0/3958914408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840c7780 con 0x7fe98406e120 2024-05-27T06:45:45.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe989600640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe97801aba0 tx=0x7fe978019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780b0040 con 0x7fe98406e120 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9780b4620 con 0x7fe98406e120 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9840c7a60 con 0x7fe98406e120 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780b4a10 con 0x7fe98406e120 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.821+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9840c8540 con 0x7fe98406e120 2024-05-27T06:45:45.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe984050870 con 0x7fe98406e120 2024-05-27T06:45:45.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe9780ab4c0 con 0x7fe98406e120 2024-05-27T06:45:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe981600640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 0x7fe964071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe978128e30 con 0x7fe98406e120 2024-05-27T06:45:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe988c00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 0x7fe964071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:45.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.825+0000 7fe988c00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 0x7fe964071940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe970006fd0 tx=0x7fe970008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:45.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.829+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe978016110 con 0x7fe98406e120 2024-05-27T06:45:45.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe98406dcf0 con 0x7fe98406e120 2024-05-27T06:45:45.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe981600640 1 -- 172.21.15.81:0/3958914408 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7fe9780f5dc0 con 0x7fe98406e120 2024-05-27T06:45:45.984 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 msgr2=0x7fe964071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 0x7fe964071940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe970006fd0 tx=0x7fe970008040 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 msgr2=0x7fe9840cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe97801aba0 tx=0x7fe978019290 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 shutdown_connections 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe9840ca2e0 0x7fe9840cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe96406f430 0x7fe964071940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe98415fdc0 0x7fe984159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 --2- 172.21.15.81:0/3958914408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe98406e120 0x7fe9840cb170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 >> 172.21.15.81:0/3958914408 conn(0x7fe984068cb0 msgr2=0x7fe9841555e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 shutdown_connections 2024-05-27T06:45:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:45.989+0000 7fe98aa00640 1 -- 172.21.15.81:0/3958914408 wait complete. 2024-05-27T06:45:46.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-27T06:45:46.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.161+0000 7f065ec00640 1 Processor -- start 2024-05-27T06:45:46.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.161+0000 7f065ec00640 1 -- start start 2024-05-27T06:45:46.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f065ec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f06580c8dd0 0x7f0658073dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f065ec00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 0x7f0658156be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f065ec00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06580c9990 con 0x7f065806d0b0 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f065ec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0658160d60 con 0x7f06580c8dd0 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f065ec00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0658160a80 con 0x7f06580c9580 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f06580c8dd0 0x7f0658073dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f06580c8dd0 0x7f0658073dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44150/0 (socket says 172.21.15.81:44150) 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 -- 172.21.15.81:0/2236038666 learned_addr learned my addr 172.21.15.81:0/2236038666 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f064f400640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 0x7f0658156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 -- 172.21.15.81:0/2236038666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 msgr2=0x7f0658156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 0x7f0658156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 -- 172.21.15.81:0/2236038666 >> v1:172.21.15.106:6789/0 conn(0x7f065806d0b0 legacy=0x7f065806d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f0657e00640 1 -- 172.21.15.81:0/2236038666 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065806c830 con 0x7f06580c8dd0 2024-05-27T06:45:46.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.165+0000 7f064f400640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 0x7f0658156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f0657e00640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f0658073dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0648009e50 tx=0x7f064801c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cd37c878e1295b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f0657400640 1 -- 172.21.15.81:0/2236038666 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06480ac020 con 0x7f06580c8dd0 2024-05-27T06:45:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f0657400640 1 -- 172.21.15.81:0/2236038666 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06480a6410 con 0x7f06580c8dd0 2024-05-27T06:45:46.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f0657400640 1 -- 172.21.15.81:0/2236038666 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06480a67e0 con 0x7f06580c8dd0 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- 172.21.15.81:0/2236038666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 msgr2=0x7f0658073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f0658073dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0648009e50 tx=0x7f064801c7c0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- 172.21.15.81:0/2236038666 shutdown_connections 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f06580c9580 0x7f0658156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 --2- 172.21.15.81:0/2236038666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f0658073dc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- 172.21.15.81:0/2236038666 >> 172.21.15.81:0/2236038666 conn(0x7f0658071850 msgr2=0x7f06580ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- 172.21.15.81:0/2236038666 shutdown_connections 2024-05-27T06:45:46.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- 172.21.15.81:0/2236038666 wait complete. 2024-05-27T06:45:46.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 Processor -- start 2024-05-27T06:45:46.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.169+0000 7f065ec00640 1 -- start start 2024-05-27T06:45:46.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 0x7f0658159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06580c9580 0x7f06580c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0658169e10 con 0x7f06580c8dd0 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0658169c90 con 0x7f06580c9580 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0658169f90 con 0x7f065806d0b0 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44162/0 (socket says 172.21.15.81:44162) 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 -- 172.21.15.81:0/1645139295 learned_addr learned my addr 172.21.15.81:0/1645139295 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f0657e00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 0x7f0658159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065cc00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06580c9580 0x7f06580c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 -- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 msgr2=0x7f0658159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 0x7f0658159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 -- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06580c9580 msgr2=0x7f06580c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06580c9580 0x7f06580c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 -- 172.21.15.81:0/1645139295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06581f6500 con 0x7f06580c8dd0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f0657e00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 0x7f0658159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f064f400640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f064000c820 tx=0x7f064000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640015070 con 0x7f06580c8dd0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0640007a30 con 0x7f06580c8dd0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0640007e40 con 0x7f06580c8dd0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06581f67e0 con 0x7f06580c8dd0 2024-05-27T06:45:46.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06580c8330 con 0x7f06580c8dd0 2024-05-27T06:45:46.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.173+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f065806a560 con 0x7f06580c8dd0 2024-05-27T06:45:46.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.177+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0640018070 con 0x7f06580c8dd0 2024-05-27T06:45:46.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.177+0000 7f0654c00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 0x7f0634071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.177+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f064008f350 con 0x7f06580c8dd0 2024-05-27T06:45:46.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.177+0000 7f0657e00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 0x7f0634071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.177+0000 7f0657e00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 0x7f0634071940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06480ad000 tx=0x7f06480a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.181+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f064005c120 con 0x7f06580c8dd0 2024-05-27T06:45:46.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.337+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f06580c8660 con 0x7f06580c8dd0 2024-05-27T06:45:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.337+0000 7f0654c00640 1 -- 172.21.15.81:0/1645139295 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) ==== 104+0+11 (secure 0 0 0) 0x7f064005ba20 con 0x7f06580c8dd0 2024-05-27T06:45:46.334 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 msgr2=0x7f0634071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 0x7f0634071940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06480ad000 tx=0x7f06480a7000 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 msgr2=0x7f065815b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f064000c820 tx=0x7f064000cce0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 shutdown_connections 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f063406f430 0x7f0634071940 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06580c9580 0x7f06580c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06580c8dd0 0x7f065815b7c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 --2- 172.21.15.81:0/1645139295 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f065806d0b0 0x7f0658159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 >> 172.21.15.81:0/1645139295 conn(0x7f0658071850 msgr2=0x7f06580c9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 shutdown_connections 2024-05-27T06:45:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.341+0000 7f065ec00640 1 -- 172.21.15.81:0/1645139295 wait complete. 2024-05-27T06:45:46.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-27T06:45:46.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 Processor -- start 2024-05-27T06:45:46.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 -- start start 2024-05-27T06:45:46.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f674415fe30 0x7f67440ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6744074ac0 0x7f6744074ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674406dff0 con 0x7f674406ea60 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6744160600 con 0x7f674415fe30 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f674ac00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f67441608e0 con 0x7f6744074ac0 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f6743e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f674415fe30 0x7f67440ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f6743e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f674415fe30 0x7f67440ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44176/0 (socket says 172.21.15.81:44176) 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f6743e00640 1 -- 172.21.15.81:0/1543300818 learned_addr learned my addr 172.21.15.81:0/1543300818 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.509+0000 7f673b400640 1 --2- 172.21.15.81:0/1543300818 >> v2:172.21.15.106:3300/0 conn(0x7f6744074ac0 0x7f6744074ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743e00640 1 -- 172.21.15.81:0/1543300818 >> v2:172.21.15.106:3300/0 conn(0x7f6744074ac0 msgr2=0x7f6744074ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:46.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743e00640 1 --2- 172.21.15.81:0/1543300818 >> v2:172.21.15.106:3300/0 conn(0x7f6744074ac0 0x7f6744074ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743e00640 1 -- 172.21.15.81:0/1543300818 >> v1:172.21.15.106:6789/0 conn(0x7f674406ea60 legacy=0x7f674406ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743e00640 1 -- 172.21.15.81:0/1543300818 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6744160bc0 con 0x7f674415fe30 2024-05-27T06:45:46.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743e00640 1 --2- 172.21.15.81:0/1543300818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674415fe30 0x7f67440ca4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f67440755c0 tx=0x7f673401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6310d4cc3fc394d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743400640 1 -- 172.21.15.81:0/1543300818 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340ab610 con 0x7f674415fe30 2024-05-27T06:45:46.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743400640 1 -- 172.21.15.81:0/1543300818 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67340b0070 con 0x7f674415fe30 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f6743400640 1 -- 172.21.15.81:0/1543300818 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340ab9d0 con 0x7f674415fe30 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f674ac00640 1 -- 172.21.15.81:0/1543300818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674415fe30 msgr2=0x7f67440ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.513+0000 7f674ac00640 1 --2- 172.21.15.81:0/1543300818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674415fe30 0x7f67440ca4f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f67440755c0 tx=0x7f673401c520 comp rx=0 tx=0).stop 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- 172.21.15.81:0/1543300818 shutdown_connections 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 --2- 172.21.15.81:0/1543300818 >> v2:172.21.15.106:3300/0 conn(0x7f6744074ac0 0x7f6744074ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 --2- 172.21.15.81:0/1543300818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674415fe30 0x7f67440ca4f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- 172.21.15.81:0/1543300818 >> 172.21.15.81:0/1543300818 conn(0x7f6744072500 msgr2=0x7f6744072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- 172.21.15.81:0/1543300818 shutdown_connections 2024-05-27T06:45:46.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- 172.21.15.81:0/1543300818 wait complete. 2024-05-27T06:45:46.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 Processor -- start 2024-05-27T06:45:46.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- start start 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6744074ac0 0x7f67440cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 0x7f674415aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67440cabb0 con 0x7f674406ea60 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f67440caa30 con 0x7f674415fe30 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f674ac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f67441699e0 con 0x7f6744074ac0 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f6743e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f6743e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44182/0 (socket says 172.21.15.81:44182) 2024-05-27T06:45:46.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.517+0000 7f6743e00640 1 -- 172.21.15.81:0/3209776169 learned_addr learned my addr 172.21.15.81:0/3209776169 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f673b400640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6744074ac0 0x7f67440cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6748c00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 0x7f674415aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 -- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6744074ac0 msgr2=0x7f67440cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6744074ac0 0x7f67440cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 -- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 msgr2=0x7f674415aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 0x7f674415aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 -- 172.21.15.81:0/3209776169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674414c150 con 0x7f674406ea60 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6748c00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 0x7f674415aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6743e00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6734097ee0 tx=0x7f673401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340be5f0 con 0x7f674406ea60 2024-05-27T06:45:46.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67340be790 con 0x7f674406ea60 2024-05-27T06:45:46.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67340beb80 con 0x7f674406ea60 2024-05-27T06:45:46.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674415d5d0 con 0x7f674406ea60 2024-05-27T06:45:46.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674414b410 con 0x7f674406ea60 2024-05-27T06:45:46.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674406a560 con 0x7f674406ea60 2024-05-27T06:45:46.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.521+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f67340b43f0 con 0x7f674406ea60 2024-05-27T06:45:46.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.525+0000 7f6740c00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 0x7f6720071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.525+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6734129b00 con 0x7f674406ea60 2024-05-27T06:45:46.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.525+0000 7f673b400640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 0x7f6720071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.525+0000 7f673b400640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 0x7f6720071940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f674415d8a0 tx=0x7f672c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.529+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f67340a8050 con 0x7f674406ea60 2024-05-27T06:45:46.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67440c52a0 con 0x7f674406ea60 2024-05-27T06:45:46.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f6740c00640 1 -- 172.21.15.81:0/3209776169 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) ==== 100+0+11 (secure 0 0 0) 0x7f6734016110 con 0x7f674406ea60 2024-05-27T06:45:46.685 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 msgr2=0x7f6720071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 0x7f6720071940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f674415d8a0 tx=0x7f672c009190 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 msgr2=0x7f674414bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6734097ee0 tx=0x7f673401ad60 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 shutdown_connections 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6744074ac0 0x7f67440cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f672006f430 0x7f6720071940 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f674415fe30 0x7f674415aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 --2- 172.21.15.81:0/3209776169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f674406ea60 0x7f674414bc10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 >> 172.21.15.81:0/3209776169 conn(0x7f6744072500 msgr2=0x7f67440ca1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 shutdown_connections 2024-05-27T06:45:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.689+0000 7f674ac00640 1 -- 172.21.15.81:0/3209776169 wait complete. 2024-05-27T06:45:46.706 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-05-27T06:45:46.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:46.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.857+0000 7f553da00640 1 Processor -- start 2024-05-27T06:45:46.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.857+0000 7f553da00640 1 -- start start 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f553da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f553da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f55380732f0 0x7f55380c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f553da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553815db00 con 0x7f55380c7510 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f553da00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f553815e6f0 con 0x7f55380732f0 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f553da00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f55380755b0 con 0x7f55381546f0 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f55380732f0 0x7f55380c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f55380732f0 0x7f55380c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44190/0 (socket says 172.21.15.81:44190) 2024-05-27T06:45:46.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 -- 172.21.15.81:0/284923993 learned_addr learned my addr 172.21.15.81:0/284923993 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f5537400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f55380c7510 0x7f553815a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34246/0 (socket says 172.21.15.81:34246) 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f5536a00640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 -- 172.21.15.81:0/284923993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 msgr2=0x7f5538072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 -- 172.21.15.81:0/284923993 >> v1:172.21.15.81:6789/0 conn(0x7f55380c7510 legacy=0x7f553815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f552e000640 1 -- 172.21.15.81:0/284923993 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5538073860 con 0x7f55380732f0 2024-05-27T06:45:46.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.861+0000 7f5536000640 1 -- 172.21.15.81:0/284923993 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2763097397 0 0) 0x7f553815db00 con 0x7f55380c7510 2024-05-27T06:45:46.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f552e000640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f55380c6fd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5520009870 tx=0x7f552001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f409fc71a72ae79 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f5536000640 1 -- 172.21.15.81:0/284923993 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55200b5070 con 0x7f55380732f0 2024-05-27T06:45:46.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f5536000640 1 -- 172.21.15.81:0/284923993 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55200a8a50 con 0x7f55380732f0 2024-05-27T06:45:46.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f5536000640 1 -- 172.21.15.81:0/284923993 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55200a8e20 con 0x7f55380732f0 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- 172.21.15.81:0/284923993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 msgr2=0x7f55380c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f55380c6fd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5520009870 tx=0x7f552001ed50 comp rx=0 tx=0).stop 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- 172.21.15.81:0/284923993 shutdown_connections 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f55380c6fd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 --2- 172.21.15.81:0/284923993 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- 172.21.15.81:0/284923993 >> 172.21.15.81:0/284923993 conn(0x7f5538070700 msgr2=0x7f55380cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- 172.21.15.81:0/284923993 shutdown_connections 2024-05-27T06:45:46.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- 172.21.15.81:0/284923993 wait complete. 2024-05-27T06:45:46.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 Processor -- start 2024-05-27T06:45:46.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.865+0000 7f553da00640 1 -- start start 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 0x7f55381f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f553815e380 con 0x7f55380732f0 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f553815e0a0 con 0x7f55380c7510 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5538154b00 con 0x7f55381546f0 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44200/0 (socket says 172.21.15.81:44200) 2024-05-27T06:45:46.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 -- 172.21.15.81:0/3644451890 learned_addr learned my addr 172.21.15.81:0/3644451890 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5537400640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f552e000640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 0x7f55381f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 -- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 msgr2=0x7f5538152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538152c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 -- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 msgr2=0x7f55381f0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 0x7f55381f0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 -- 172.21.15.81:0/3644451890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55381f0fe0 con 0x7f55380732f0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f552e000640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 0x7f55381f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f5536a00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f552800e870 tx=0x7f552800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528017070 con 0x7f55380732f0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552800c940 con 0x7f55380732f0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552800ccf0 con 0x7f55380732f0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5538158d60 con 0x7f55380732f0 2024-05-27T06:45:46.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55381f43a0 con 0x7f55380732f0 2024-05-27T06:45:46.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.869+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f553806a560 con 0x7f55380732f0 2024-05-27T06:45:46.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.873+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f552801a070 con 0x7f55380732f0 2024-05-27T06:45:46.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.873+0000 7f552f400640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 0x7f5514071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:46.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.873+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f552808f510 con 0x7f55380732f0 2024-05-27T06:45:46.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.873+0000 7f552e000640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 0x7f5514071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:46.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.873+0000 7f552e000640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 0x7f5514071940 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f55200b2000 tx=0x7f5520008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:46.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:46.877+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f552805bd20 con 0x7f55380732f0 2024-05-27T06:45:47.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5538050d90 con 0x7f55380732f0 2024-05-27T06:45:47.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f552f400640 1 -- 172.21.15.81:0/3644451890 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f5528062070 con 0x7f55380732f0 2024-05-27T06:45:47.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:47.065 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:45:47.065 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 msgr2=0x7f5514071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 0x7f5514071940 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f55200b2000 tx=0x7f5520008040 comp rx=0 tx=0).stop 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 msgr2=0x7f5538155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f552800e870 tx=0x7f552800ec70 comp rx=0 tx=0).stop 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 shutdown_connections 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f55381546f0 0x7f5538152c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f551406f430 0x7f5514071940 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f55380c7510 0x7f55381f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 --2- 172.21.15.81:0/3644451890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55380732f0 0x7f5538155960 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 >> 172.21.15.81:0/3644451890 conn(0x7f5538070700 msgr2=0x7f5538153830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:47.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 shutdown_connections 2024-05-27T06:45:47.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.069+0000 7f553da00640 1 -- 172.21.15.81:0/3644451890 wait complete. 2024-05-27T06:45:47.086 INFO:tasks.mds_thrash:Ready to start thrashing 2024-05-27T06:45:47.086 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-05-27T06:45:47.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:47.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.237+0000 7ff9ba200640 1 Processor -- start 2024-05-27T06:45:47.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.237+0000 7ff9ba200640 1 -- start start 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9b406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9ba200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b40c51d0 con 0x7ff9b40c98f0 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9ba200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b40c8440 con 0x7ff9b40c4dc0 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9ba200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b40c5450 con 0x7ff9b40c94e0 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff9b40c98f0 0x7ff9b416b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34262/0 (socket says 172.21.15.81:34262) 2024-05-27T06:45:47.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 -- 172.21.15.81:0/4220446785 learned_addr learned my addr 172.21.15.81:0/4220446785 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9aaa00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9b406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2642742319 0 0) 0x7ff9ac001120 con 0x7ff9b40c98f0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff998003530 con 0x7ff9b40c98f0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 132913685 0 0) 0x7ff998003530 con 0x7ff9b40c98f0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 msgr2=0x7ff9b406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9b406c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 >> v1:172.21.15.106:6789/0 conn(0x7ff9b40c4dc0 legacy=0x7ff9b4164af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b40c8720 con 0x7ff9b40c98f0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7ff9ac0018d0 con 0x7ff9b40c98f0 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 >> v1:172.21.15.81:6789/0 conn(0x7ff9b40c98f0 legacy=0x7ff9b416b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9980039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998003fa0 0x7ff998004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff998004d50 0x7ff9980051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b40c5450 con 0x7ff998004d50 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff99800bb90 con 0x7ff998003fa0 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff99800bd90 con 0x7ff9b40c94e0 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff998004d50 0x7ff9980051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9aaa00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998003fa0 0x7ff998004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3400640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9980039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 -- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 msgr2=0x7ff9980039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b40c94e0 0x7ff9980039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 -- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998003fa0 msgr2=0x7ff998004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998003fa0 0x7ff998004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9b3e00640 1 -- 172.21.15.81:0/4220446785 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff998003530 con 0x7ff998004d50 2024-05-27T06:45:47.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.241+0000 7ff9aaa00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998003fa0 0x7ff998004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:47.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff998004d50 0x7ff9980051a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c490 tx=0x7ff9ac01f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6489ecfc24252437 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00c0e0 con 0x7ff998004d50 2024-05-27T06:45:47.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac005b00 con 0x7ff998004d50 2024-05-27T06:45:47.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9b2a00640 1 -- 172.21.15.81:0/4220446785 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac007070 con 0x7ff998004d50 2024-05-27T06:45:47.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 -- 172.21.15.81:0/4220446785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff998004d50 msgr2=0x7ff9980051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 --2- 172.21.15.81:0/4220446785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff998004d50 0x7ff9980051a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c490 tx=0x7ff9ac01f2d0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9b3e00640 1 -- 172.21.15.81:0/4220446785 reap_dead start 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 -- 172.21.15.81:0/4220446785 shutdown_connections 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 -- 172.21.15.81:0/4220446785 >> 172.21.15.81:0/4220446785 conn(0x7ff9b4070700 msgr2=0x7ff9b40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 -- 172.21.15.81:0/4220446785 shutdown_connections 2024-05-27T06:45:47.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.245+0000 7ff9ba200640 1 -- 172.21.15.81:0/4220446785 wait complete. 2024-05-27T06:45:47.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 Processor -- start 2024-05-27T06:45:47.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- start start 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998004d50 0x7ff9b41f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b4162c60 0x7ff9b4154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b40c8fc0 con 0x7ff9b4153e70 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b40c8ce0 con 0x7ff998004d50 2024-05-27T06:45:47.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b40ca050 con 0x7ff9b4162c60 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44212/0 (socket says 172.21.15.81:44212) 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 -- 172.21.15.81:0/1924820648 learned_addr learned my addr 172.21.15.81:0/1924820648 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9aaa00640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b4162c60 0x7ff9b4154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3400640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998004d50 0x7ff9b41f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 -- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b4162c60 msgr2=0x7ff9b4154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b4162c60 0x7ff9b4154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 -- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998004d50 msgr2=0x7ff9b41f7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998004d50 0x7ff9b41f7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 -- 172.21.15.81:0/1924820648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b4161ea0 con 0x7ff9b4153e70 2024-05-27T06:45:47.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9b3e00640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c490 tx=0x7ff998005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0b4030 con 0x7ff9b4153e70 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac001120 con 0x7ff9b4153e70 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b4162180 con 0x7ff9b4153e70 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0096b0 con 0x7ff9b4153e70 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.249+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b4154ed0 con 0x7ff9b4153e70 2024-05-27T06:45:47.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b4155220 con 0x7ff9b4153e70 2024-05-27T06:45:47.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff9ac0ba030 con 0x7ff9b4153e70 2024-05-27T06:45:47.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9abe00640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 0x7ff990071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff9ac12bb30 con 0x7ff9b4153e70 2024-05-27T06:45:47.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9b3400640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 0x7ff990071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.253+0000 7ff9b3400640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 0x7ff990071910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4001530 tx=0x7ff9a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.257+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff9b40c51d0 con 0x7ff9b4153e70 2024-05-27T06:45:47.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.413+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9b406a560 con 0x7ff99006f400 2024-05-27T06:45:47.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.417+0000 7ff9abe00640 1 -- 172.21.15.81:0/1924820648 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251984 (secure 0 0 0) 0x7ff9b406a560 con 0x7ff99006f400 2024-05-27T06:45:47.417 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:47.419 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:47.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 msgr2=0x7ff990071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 0x7ff990071910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4001530 tx=0x7ff9a4009040 comp rx=0 tx=0).stop 2024-05-27T06:45:47.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 msgr2=0x7ff9b40744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00c490 tx=0x7ff998005840 comp rx=0 tx=0).stop 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 shutdown_connections 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff9b4162c60 0x7ff9b4154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff99006f400 0x7ff990071910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9b4153e70 0x7ff9b40744d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 --2- 172.21.15.81:0/1924820648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff998004d50 0x7ff9b41f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 >> 172.21.15.81:0/1924820648 conn(0x7ff9b4070700 msgr2=0x7ff9b414fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 shutdown_connections 2024-05-27T06:45:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.421+0000 7ff9ba200640 1 -- 172.21.15.81:0/1924820648 wait complete. 2024-05-27T06:45:47.438 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-05-27T06:45:45.501144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":306,"num_write_kb":1690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":251,"ondisk_log_size":251,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218208,"kb_used_data":2080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756512,"statfs":{"total":773094113280,"available":772870668288,"internally_reserved":0,"allocated":2129920,"data_stored":1801644,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001720"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620285+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:03.620285+0000","last_peered":"2024-05-27T06:45:03.620285+0000","last_clean":"2024-05-27T06:45:03.620285+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:03.620285+0000","last_undegraded":"2024-05-27T06:45:03.620285+0000","last_fullsized":"2024-05-27T06:45:03.620285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617672+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:03.617672+0000","last_peered":"2024-05-27T06:45:03.617672+0000","last_clean":"2024-05-27T06:45:03.617672+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:03.617672+0000","last_undegraded":"2024-05-27T06:45:03.617672+0000","last_fullsized":"2024-05-27T06:45:03.617672+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542582+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:04.542582+0000","last_peered":"2024-05-27T06:45:04.542582+0000","last_clean":"2024-05-27T06:45:04.542582+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:04.542582+0000","last_undegraded":"2024-05-27T06:45:04.542582+0000","last_fullsized":"2024-05-27T06:45:04.542582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619261+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:03.619261+0000","last_peered":"2024-05-27T06:45:03.619261+0000","last_clean":"2024-05-27T06:45:03.619261+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:03.619261+0000","last_undegraded":"2024-05-27T06:45:03.619261+0000","last_fullsized":"2024-05-27T06:45:03.619261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620460+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:03.620460+0000","last_peered":"2024-05-27T06:45:03.620460+0000","last_clean":"2024-05-27T06:45:03.620460+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:03.620460+0000","last_undegraded":"2024-05-27T06:45:03.620460+0000","last_fullsized":"2024-05-27T06:45:03.620460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619485+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:03.619485+0000","last_peered":"2024-05-27T06:45:03.619485+0000","last_clean":"2024-05-27T06:45:03.619485+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:03.619485+0000","last_undegraded":"2024-05-27T06:45:03.619485+0000","last_fullsized":"2024-05-27T06:45:03.619485+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618534+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:03.618534+0000","last_peered":"2024-05-27T06:45:03.618534+0000","last_clean":"2024-05-27T06:45:03.618534+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:03.618534+0000","last_undegraded":"2024-05-27T06:45:03.618534+0000","last_fullsized":"2024-05-27T06:45:03.618534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542255+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:04.542255+0000","last_peered":"2024-05-27T06:45:04.542255+0000","last_clean":"2024-05-27T06:45:04.542255+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:04.542255+0000","last_undegraded":"2024-05-27T06:45:04.542255+0000","last_fullsized":"2024-05-27T06:45:04.542255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620365+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:03.620365+0000","last_peered":"2024-05-27T06:45:03.620365+0000","last_clean":"2024-05-27T06:45:03.620365+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:03.620365+0000","last_undegraded":"2024-05-27T06:45:03.620365+0000","last_fullsized":"2024-05-27T06:45:03.620365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542559+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:04.542559+0000","last_peered":"2024-05-27T06:45:04.542559+0000","last_clean":"2024-05-27T06:45:04.542559+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:04.542559+0000","last_undegraded":"2024-05-27T06:45:04.542559+0000","last_fullsized":"2024-05-27T06:45:04.542559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542270+0000","last_change":"2024-05-27T06:45:02.647403+0000","last_active":"2024-05-27T06:45:04.542270+0000","last_peered":"2024-05-27T06:45:04.542270+0000","last_clean":"2024-05-27T06:45:04.542270+0000","last_became_active":"2024-05-27T06:45:02.647061+0000","last_became_peered":"2024-05-27T06:45:02.647061+0000","last_unstale":"2024-05-27T06:45:04.542270+0000","last_undegraded":"2024-05-27T06:45:04.542270+0000","last_fullsized":"2024-05-27T06:45:04.542270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:10:58.239761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543063+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:04.543063+0000","last_peered":"2024-05-27T06:45:04.543063+0000","last_clean":"2024-05-27T06:45:04.543063+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:04.543063+0000","last_undegraded":"2024-05-27T06:45:04.543063+0000","last_fullsized":"2024-05-27T06:45:04.543063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542091+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:04.542091+0000","last_peered":"2024-05-27T06:45:04.542091+0000","last_clean":"2024-05-27T06:45:04.542091+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:04.542091+0000","last_undegraded":"2024-05-27T06:45:04.542091+0000","last_fullsized":"2024-05-27T06:45:04.542091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542474+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:04.542474+0000","last_peered":"2024-05-27T06:45:04.542474+0000","last_clean":"2024-05-27T06:45:04.542474+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:04.542474+0000","last_undegraded":"2024-05-27T06:45:04.542474+0000","last_fullsized":"2024-05-27T06:45:04.542474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617212+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:03.617212+0000","last_peered":"2024-05-27T06:45:03.617212+0000","last_clean":"2024-05-27T06:45:03.617212+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:03.617212+0000","last_undegraded":"2024-05-27T06:45:03.617212+0000","last_fullsized":"2024-05-27T06:45:03.617212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619654+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:03.619654+0000","last_peered":"2024-05-27T06:45:03.619654+0000","last_clean":"2024-05-27T06:45:03.619654+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:03.619654+0000","last_undegraded":"2024-05-27T06:45:03.619654+0000","last_fullsized":"2024-05-27T06:45:03.619654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620222+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:03.620222+0000","last_peered":"2024-05-27T06:45:03.620222+0000","last_clean":"2024-05-27T06:45:03.620222+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:03.620222+0000","last_undegraded":"2024-05-27T06:45:03.620222+0000","last_fullsized":"2024-05-27T06:45:03.620222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541770+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:04.541770+0000","last_peered":"2024-05-27T06:45:04.541770+0000","last_clean":"2024-05-27T06:45:04.541770+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:04.541770+0000","last_undegraded":"2024-05-27T06:45:04.541770+0000","last_fullsized":"2024-05-27T06:45:04.541770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620170+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:03.620170+0000","last_peered":"2024-05-27T06:45:03.620170+0000","last_clean":"2024-05-27T06:45:03.620170+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:03.620170+0000","last_undegraded":"2024-05-27T06:45:03.620170+0000","last_fullsized":"2024-05-27T06:45:03.620170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542157+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:04.542157+0000","last_peered":"2024-05-27T06:45:04.542157+0000","last_clean":"2024-05-27T06:45:04.542157+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:04.542157+0000","last_undegraded":"2024-05-27T06:45:04.542157+0000","last_fullsized":"2024-05-27T06:45:04.542157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542660+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:04.542660+0000","last_peered":"2024-05-27T06:45:04.542660+0000","last_clean":"2024-05-27T06:45:04.542660+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:04.542660+0000","last_undegraded":"2024-05-27T06:45:04.542660+0000","last_fullsized":"2024-05-27T06:45:04.542660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619623+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:03.619623+0000","last_peered":"2024-05-27T06:45:03.619623+0000","last_clean":"2024-05-27T06:45:03.619623+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:03.619623+0000","last_undegraded":"2024-05-27T06:45:03.619623+0000","last_fullsized":"2024-05-27T06:45:03.619623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542082+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:04.542082+0000","last_peered":"2024-05-27T06:45:04.542082+0000","last_clean":"2024-05-27T06:45:04.542082+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:04.542082+0000","last_undegraded":"2024-05-27T06:45:04.542082+0000","last_fullsized":"2024-05-27T06:45:04.542082+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619147+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:03.619147+0000","last_peered":"2024-05-27T06:45:03.619147+0000","last_clean":"2024-05-27T06:45:03.619147+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:03.619147+0000","last_undegraded":"2024-05-27T06:45:03.619147+0000","last_fullsized":"2024-05-27T06:45:03.619147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543112+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:04.543112+0000","last_peered":"2024-05-27T06:45:04.543112+0000","last_clean":"2024-05-27T06:45:04.543112+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:04.543112+0000","last_undegraded":"2024-05-27T06:45:04.543112+0000","last_fullsized":"2024-05-27T06:45:04.543112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542613+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:04.542613+0000","last_peered":"2024-05-27T06:45:04.542613+0000","last_clean":"2024-05-27T06:45:04.542613+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:04.542613+0000","last_undegraded":"2024-05-27T06:45:04.542613+0000","last_fullsized":"2024-05-27T06:45:04.542613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617693+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:03.617693+0000","last_peered":"2024-05-27T06:45:03.617693+0000","last_clean":"2024-05-27T06:45:03.617693+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:03.617693+0000","last_undegraded":"2024-05-27T06:45:03.617693+0000","last_fullsized":"2024-05-27T06:45:03.617693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541962+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:04.541962+0000","last_peered":"2024-05-27T06:45:04.541962+0000","last_clean":"2024-05-27T06:45:04.541962+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:04.541962+0000","last_undegraded":"2024-05-27T06:45:04.541962+0000","last_fullsized":"2024-05-27T06:45:04.541962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542858+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:04.542858+0000","last_peered":"2024-05-27T06:45:04.542858+0000","last_clean":"2024-05-27T06:45:04.542858+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:04.542858+0000","last_undegraded":"2024-05-27T06:45:04.542858+0000","last_fullsized":"2024-05-27T06:45:04.542858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619714+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:03.619714+0000","last_peered":"2024-05-27T06:45:03.619714+0000","last_clean":"2024-05-27T06:45:03.619714+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:03.619714+0000","last_undegraded":"2024-05-27T06:45:03.619714+0000","last_fullsized":"2024-05-27T06:45:03.619714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542030+0000","last_change":"2024-05-27T06:45:02.627394+0000","last_active":"2024-05-27T06:45:04.542030+0000","last_peered":"2024-05-27T06:45:04.542030+0000","last_clean":"2024-05-27T06:45:04.542030+0000","last_became_active":"2024-05-27T06:45:02.626550+0000","last_became_peered":"2024-05-27T06:45:02.626550+0000","last_unstale":"2024-05-27T06:45:04.542030+0000","last_undegraded":"2024-05-27T06:45:04.542030+0000","last_fullsized":"2024-05-27T06:45:04.542030+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:03:01.383493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541845+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:04.541845+0000","last_peered":"2024-05-27T06:45:04.541845+0000","last_clean":"2024-05-27T06:45:04.541845+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:04.541845+0000","last_undegraded":"2024-05-27T06:45:04.541845+0000","last_fullsized":"2024-05-27T06:45:04.541845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542588+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:04.542588+0000","last_peered":"2024-05-27T06:45:04.542588+0000","last_clean":"2024-05-27T06:45:04.542588+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:04.542588+0000","last_undegraded":"2024-05-27T06:45:04.542588+0000","last_fullsized":"2024-05-27T06:45:04.542588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:16.687487+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:16.687487+0000","last_peered":"2024-05-27T06:45:16.687487+0000","last_clean":"2024-05-27T06:45:16.687487+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:16.687487+0000","last_undegraded":"2024-05-27T06:45:16.687487+0000","last_fullsized":"2024-05-27T06:45:16.687487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543053+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:04.543053+0000","last_peered":"2024-05-27T06:45:04.543053+0000","last_clean":"2024-05-27T06:45:04.543053+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:04.543053+0000","last_undegraded":"2024-05-27T06:45:04.543053+0000","last_fullsized":"2024-05-27T06:45:04.543053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.221957+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:28.221957+0000","last_peered":"2024-05-27T06:45:28.221957+0000","last_clean":"2024-05-27T06:45:28.221957+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:28.221957+0000","last_undegraded":"2024-05-27T06:45:28.221957+0000","last_fullsized":"2024-05-27T06:45:28.221957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:21.885674+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:21.885674+0000","last_peered":"2024-05-27T06:45:21.885674+0000","last_clean":"2024-05-27T06:45:21.885674+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:21.885674+0000","last_undegraded":"2024-05-27T06:45:21.885674+0000","last_fullsized":"2024-05-27T06:45:21.885674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542424+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:04.542424+0000","last_peered":"2024-05-27T06:45:04.542424+0000","last_clean":"2024-05-27T06:45:04.542424+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:04.542424+0000","last_undegraded":"2024-05-27T06:45:04.542424+0000","last_fullsized":"2024-05-27T06:45:04.542424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.218942+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:28.218942+0000","last_peered":"2024-05-27T06:45:28.218942+0000","last_clean":"2024-05-27T06:45:28.218942+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:28.218942+0000","last_undegraded":"2024-05-27T06:45:28.218942+0000","last_fullsized":"2024-05-27T06:45:28.218942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617259+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:03.617259+0000","last_peered":"2024-05-27T06:45:03.617259+0000","last_clean":"2024-05-27T06:45:03.617259+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:03.617259+0000","last_undegraded":"2024-05-27T06:45:03.617259+0000","last_fullsized":"2024-05-27T06:45:03.617259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:16.686777+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:16.686777+0000","last_peered":"2024-05-27T06:45:16.686777+0000","last_clean":"2024-05-27T06:45:16.686777+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:16.686777+0000","last_undegraded":"2024-05-27T06:45:16.686777+0000","last_fullsized":"2024-05-27T06:45:16.686777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619378+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:03.619378+0000","last_peered":"2024-05-27T06:45:03.619378+0000","last_clean":"2024-05-27T06:45:03.619378+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:03.619378+0000","last_undegraded":"2024-05-27T06:45:03.619378+0000","last_fullsized":"2024-05-27T06:45:03.619378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.611197+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:33.611197+0000","last_peered":"2024-05-27T06:45:33.611197+0000","last_clean":"2024-05-27T06:45:33.611197+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:33.611197+0000","last_undegraded":"2024-05-27T06:45:33.611197+0000","last_fullsized":"2024-05-27T06:45:33.611197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543182+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:04.543182+0000","last_peered":"2024-05-27T06:45:04.543182+0000","last_clean":"2024-05-27T06:45:04.543182+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:04.543182+0000","last_undegraded":"2024-05-27T06:45:04.543182+0000","last_fullsized":"2024-05-27T06:45:04.543182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:16.686265+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:16.686265+0000","last_peered":"2024-05-27T06:45:16.686265+0000","last_clean":"2024-05-27T06:45:16.686265+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:16.686265+0000","last_undegraded":"2024-05-27T06:45:16.686265+0000","last_fullsized":"2024-05-27T06:45:16.686265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543150+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:04.543150+0000","last_peered":"2024-05-27T06:45:04.543150+0000","last_clean":"2024-05-27T06:45:04.543150+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:04.543150+0000","last_undegraded":"2024-05-27T06:45:04.543150+0000","last_fullsized":"2024-05-27T06:45:04.543150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.613106+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:33.613106+0000","last_peered":"2024-05-27T06:45:33.613106+0000","last_clean":"2024-05-27T06:45:33.613106+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:33.613106+0000","last_undegraded":"2024-05-27T06:45:33.613106+0000","last_fullsized":"2024-05-27T06:45:33.613106+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618838+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:03.618838+0000","last_peered":"2024-05-27T06:45:03.618838+0000","last_clean":"2024-05-27T06:45:03.618838+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:03.618838+0000","last_undegraded":"2024-05-27T06:45:03.618838+0000","last_fullsized":"2024-05-27T06:45:03.618838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.609811+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:33.609811+0000","last_peered":"2024-05-27T06:45:33.609811+0000","last_clean":"2024-05-27T06:45:33.609811+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:33.609811+0000","last_undegraded":"2024-05-27T06:45:33.609811+0000","last_fullsized":"2024-05-27T06:45:33.609811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542669+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:04.542669+0000","last_peered":"2024-05-27T06:45:04.542669+0000","last_clean":"2024-05-27T06:45:04.542669+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:04.542669+0000","last_undegraded":"2024-05-27T06:45:04.542669+0000","last_fullsized":"2024-05-27T06:45:04.542669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.217769+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:28.217769+0000","last_peered":"2024-05-27T06:45:28.217769+0000","last_clean":"2024-05-27T06:45:28.217769+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:28.217769+0000","last_undegraded":"2024-05-27T06:45:28.217769+0000","last_fullsized":"2024-05-27T06:45:28.217769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619590+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:03.619590+0000","last_peered":"2024-05-27T06:45:03.619590+0000","last_clean":"2024-05-27T06:45:03.619590+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:03.619590+0000","last_undegraded":"2024-05-27T06:45:03.619590+0000","last_fullsized":"2024-05-27T06:45:03.619590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.219191+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:28.219191+0000","last_peered":"2024-05-27T06:45:28.219191+0000","last_clean":"2024-05-27T06:45:28.219191+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:28.219191+0000","last_undegraded":"2024-05-27T06:45:28.219191+0000","last_fullsized":"2024-05-27T06:45:28.219191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541996+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:04.541996+0000","last_peered":"2024-05-27T06:45:04.541996+0000","last_clean":"2024-05-27T06:45:04.541996+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:04.541996+0000","last_undegraded":"2024-05-27T06:45:04.541996+0000","last_fullsized":"2024-05-27T06:45:04.541996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.215094+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:28.215094+0000","last_peered":"2024-05-27T06:45:28.215094+0000","last_clean":"2024-05-27T06:45:28.215094+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:28.215094+0000","last_undegraded":"2024-05-27T06:45:28.215094+0000","last_fullsized":"2024-05-27T06:45:28.215094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619502+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:03.619502+0000","last_peered":"2024-05-27T06:45:03.619502+0000","last_clean":"2024-05-27T06:45:03.619502+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:03.619502+0000","last_undegraded":"2024-05-27T06:45:03.619502+0000","last_fullsized":"2024-05-27T06:45:03.619502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.640819+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:07.640819+0000","last_peered":"2024-05-27T06:45:07.640819+0000","last_clean":"2024-05-27T06:45:07.640819+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:07.640819+0000","last_undegraded":"2024-05-27T06:45:07.640819+0000","last_fullsized":"2024-05-27T06:45:07.640819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620404+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:03.620404+0000","last_peered":"2024-05-27T06:45:03.620404+0000","last_clean":"2024-05-27T06:45:03.620404+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:03.620404+0000","last_undegraded":"2024-05-27T06:45:03.620404+0000","last_fullsized":"2024-05-27T06:45:03.620404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.608469+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:33.608469+0000","last_peered":"2024-05-27T06:45:33.608469+0000","last_clean":"2024-05-27T06:45:33.608469+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:33.608469+0000","last_undegraded":"2024-05-27T06:45:33.608469+0000","last_fullsized":"2024-05-27T06:45:33.608469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620405+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:03.620405+0000","last_peered":"2024-05-27T06:45:03.620405+0000","last_clean":"2024-05-27T06:45:03.620405+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:03.620405+0000","last_undegraded":"2024-05-27T06:45:03.620405+0000","last_fullsized":"2024-05-27T06:45:03.620405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.420103+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:28.420103+0000","last_peered":"2024-05-27T06:45:28.420103+0000","last_clean":"2024-05-27T06:45:28.420103+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:28.420103+0000","last_undegraded":"2024-05-27T06:45:28.420103+0000","last_fullsized":"2024-05-27T06:45:28.420103+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619736+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:03.619736+0000","last_peered":"2024-05-27T06:45:03.619736+0000","last_clean":"2024-05-27T06:45:03.619736+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:03.619736+0000","last_undegraded":"2024-05-27T06:45:03.619736+0000","last_fullsized":"2024-05-27T06:45:03.619736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.219407+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:28.219407+0000","last_peered":"2024-05-27T06:45:28.219407+0000","last_clean":"2024-05-27T06:45:28.219407+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:28.219407+0000","last_undegraded":"2024-05-27T06:45:28.219407+0000","last_fullsized":"2024-05-27T06:45:28.219407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.620105+0000","last_change":"2024-05-27T06:45:02.651256+0000","last_active":"2024-05-27T06:45:03.620105+0000","last_peered":"2024-05-27T06:45:03.620105+0000","last_clean":"2024-05-27T06:45:03.620105+0000","last_became_active":"2024-05-27T06:45:02.650646+0000","last_became_peered":"2024-05-27T06:45:02.650646+0000","last_unstale":"2024-05-27T06:45:03.620105+0000","last_undegraded":"2024-05-27T06:45:03.620105+0000","last_fullsized":"2024-05-27T06:45:03.620105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:39:57.469688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.618689+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:03.618689+0000","last_peered":"2024-05-27T06:45:03.618689+0000","last_clean":"2024-05-27T06:45:03.618689+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:03.618689+0000","last_undegraded":"2024-05-27T06:45:03.618689+0000","last_fullsized":"2024-05-27T06:45:03.618689+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.218862+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:28.218862+0000","last_peered":"2024-05-27T06:45:28.218862+0000","last_clean":"2024-05-27T06:45:28.218862+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:28.218862+0000","last_undegraded":"2024-05-27T06:45:28.218862+0000","last_fullsized":"2024-05-27T06:45:28.218862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541730+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:04.541730+0000","last_peered":"2024-05-27T06:45:04.541730+0000","last_clean":"2024-05-27T06:45:04.541730+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:04.541730+0000","last_undegraded":"2024-05-27T06:45:04.541730+0000","last_fullsized":"2024-05-27T06:45:04.541730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:21.887794+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:21.887794+0000","last_peered":"2024-05-27T06:45:21.887794+0000","last_clean":"2024-05-27T06:45:21.887794+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:21.887794+0000","last_undegraded":"2024-05-27T06:45:21.887794+0000","last_fullsized":"2024-05-27T06:45:21.887794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543183+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:04.543183+0000","last_peered":"2024-05-27T06:45:04.543183+0000","last_clean":"2024-05-27T06:45:04.543183+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:04.543183+0000","last_undegraded":"2024-05-27T06:45:04.543183+0000","last_fullsized":"2024-05-27T06:45:04.543183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:21.886725+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:21.886725+0000","last_peered":"2024-05-27T06:45:21.886725+0000","last_clean":"2024-05-27T06:45:21.886725+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:21.886725+0000","last_undegraded":"2024-05-27T06:45:21.886725+0000","last_fullsized":"2024-05-27T06:45:21.886725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617210+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:03.617210+0000","last_peered":"2024-05-27T06:45:03.617210+0000","last_clean":"2024-05-27T06:45:03.617210+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:03.617210+0000","last_undegraded":"2024-05-27T06:45:03.617210+0000","last_fullsized":"2024-05-27T06:45:03.617210+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.615067+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:33.615067+0000","last_peered":"2024-05-27T06:45:33.615067+0000","last_clean":"2024-05-27T06:45:33.615067+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:33.615067+0000","last_undegraded":"2024-05-27T06:45:33.615067+0000","last_fullsized":"2024-05-27T06:45:33.615067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542806+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:04.542806+0000","last_peered":"2024-05-27T06:45:04.542806+0000","last_clean":"2024-05-27T06:45:04.542806+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:04.542806+0000","last_undegraded":"2024-05-27T06:45:04.542806+0000","last_fullsized":"2024-05-27T06:45:04.542806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617480+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:03.617480+0000","last_peered":"2024-05-27T06:45:03.617480+0000","last_clean":"2024-05-27T06:45:03.617480+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:03.617480+0000","last_undegraded":"2024-05-27T06:45:03.617480+0000","last_fullsized":"2024-05-27T06:45:03.617480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619493+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:03.619493+0000","last_peered":"2024-05-27T06:45:03.619493+0000","last_clean":"2024-05-27T06:45:03.619493+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:03.619493+0000","last_undegraded":"2024-05-27T06:45:03.619493+0000","last_fullsized":"2024-05-27T06:45:03.619493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.219357+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:28.219357+0000","last_peered":"2024-05-27T06:45:28.219357+0000","last_clean":"2024-05-27T06:45:28.219357+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:28.219357+0000","last_undegraded":"2024-05-27T06:45:28.219357+0000","last_fullsized":"2024-05-27T06:45:28.219357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.617526+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:03.617526+0000","last_peered":"2024-05-27T06:45:03.617526+0000","last_clean":"2024-05-27T06:45:03.617526+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:03.617526+0000","last_undegraded":"2024-05-27T06:45:03.617526+0000","last_fullsized":"2024-05-27T06:45:03.617526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.422156+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:28.422156+0000","last_peered":"2024-05-27T06:45:28.422156+0000","last_clean":"2024-05-27T06:45:28.422156+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:28.422156+0000","last_undegraded":"2024-05-27T06:45:28.422156+0000","last_fullsized":"2024-05-27T06:45:28.422156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543116+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:04.543116+0000","last_peered":"2024-05-27T06:45:04.543116+0000","last_clean":"2024-05-27T06:45:04.543116+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:04.543116+0000","last_undegraded":"2024-05-27T06:45:04.543116+0000","last_fullsized":"2024-05-27T06:45:04.543116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.613012+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:33.613012+0000","last_peered":"2024-05-27T06:45:33.613012+0000","last_clean":"2024-05-27T06:45:33.613012+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:33.613012+0000","last_undegraded":"2024-05-27T06:45:33.613012+0000","last_fullsized":"2024-05-27T06:45:33.613012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542819+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:04.542819+0000","last_peered":"2024-05-27T06:45:04.542819+0000","last_clean":"2024-05-27T06:45:04.542819+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:04.542819+0000","last_undegraded":"2024-05-27T06:45:04.542819+0000","last_fullsized":"2024-05-27T06:45:04.542819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:16.683562+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:16.683562+0000","last_peered":"2024-05-27T06:45:16.683562+0000","last_clean":"2024-05-27T06:45:16.683562+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:16.683562+0000","last_undegraded":"2024-05-27T06:45:16.683562+0000","last_fullsized":"2024-05-27T06:45:16.683562+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542399+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:04.542399+0000","last_peered":"2024-05-27T06:45:04.542399+0000","last_clean":"2024-05-27T06:45:04.542399+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:04.542399+0000","last_undegraded":"2024-05-27T06:45:04.542399+0000","last_fullsized":"2024-05-27T06:45:04.542399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.611216+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:33.611216+0000","last_peered":"2024-05-27T06:45:33.611216+0000","last_clean":"2024-05-27T06:45:33.611216+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:33.611216+0000","last_undegraded":"2024-05-27T06:45:33.611216+0000","last_fullsized":"2024-05-27T06:45:33.611216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542603+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:04.542603+0000","last_peered":"2024-05-27T06:45:04.542603+0000","last_clean":"2024-05-27T06:45:04.542603+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:04.542603+0000","last_undegraded":"2024-05-27T06:45:04.542603+0000","last_fullsized":"2024-05-27T06:45:04.542603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:28.419255+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:28.419255+0000","last_peered":"2024-05-27T06:45:28.419255+0000","last_clean":"2024-05-27T06:45:28.419255+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:28.419255+0000","last_undegraded":"2024-05-27T06:45:28.419255+0000","last_fullsized":"2024-05-27T06:45:28.419255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.543008+0000","last_change":"2024-05-27T06:45:02.632527+0000","last_active":"2024-05-27T06:45:04.543008+0000","last_peered":"2024-05-27T06:45:04.543008+0000","last_clean":"2024-05-27T06:45:04.543008+0000","last_became_active":"2024-05-27T06:45:02.632140+0000","last_became_peered":"2024-05-27T06:45:02.632140+0000","last_unstale":"2024-05-27T06:45:04.543008+0000","last_undegraded":"2024-05-27T06:45:04.543008+0000","last_fullsized":"2024-05-27T06:45:04.543008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:16.685671+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:16.685671+0000","last_peered":"2024-05-27T06:45:16.685671+0000","last_clean":"2024-05-27T06:45:16.685671+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:16.685671+0000","last_undegraded":"2024-05-27T06:45:16.685671+0000","last_fullsized":"2024-05-27T06:45:16.685671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541876+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:04.541876+0000","last_peered":"2024-05-27T06:45:04.541876+0000","last_clean":"2024-05-27T06:45:04.541876+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:04.541876+0000","last_undegraded":"2024-05-27T06:45:04.541876+0000","last_fullsized":"2024-05-27T06:45:04.541876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619831+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:03.619831+0000","last_peered":"2024-05-27T06:45:03.619831+0000","last_clean":"2024-05-27T06:45:03.619831+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:03.619831+0000","last_undegraded":"2024-05-27T06:45:03.619831+0000","last_fullsized":"2024-05-27T06:45:03.619831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.613853+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:33.613853+0000","last_peered":"2024-05-27T06:45:33.613853+0000","last_clean":"2024-05-27T06:45:33.613853+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:33.613853+0000","last_undegraded":"2024-05-27T06:45:33.613853+0000","last_fullsized":"2024-05-27T06:45:33.613853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:03.619460+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:03.619460+0000","last_peered":"2024-05-27T06:45:03.619460+0000","last_clean":"2024-05-27T06:45:03.619460+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:03.619460+0000","last_undegraded":"2024-05-27T06:45:03.619460+0000","last_fullsized":"2024-05-27T06:45:03.619460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":26,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.614723+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:33.614723+0000","last_peered":"2024-05-27T06:45:33.614723+0000","last_clean":"2024-05-27T06:45:33.614723+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:33.614723+0000","last_undegraded":"2024-05-27T06:45:33.614723+0000","last_fullsized":"2024-05-27T06:45:33.614723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.542992+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:04.542992+0000","last_peered":"2024-05-27T06:45:04.542992+0000","last_clean":"2024-05-27T06:45:04.542992+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:04.542992+0000","last_undegraded":"2024-05-27T06:45:04.542992+0000","last_fullsized":"2024-05-27T06:45:04.542992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:07.642764+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:07.642764+0000","last_peered":"2024-05-27T06:45:07.642764+0000","last_clean":"2024-05-27T06:45:07.642764+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:07.642764+0000","last_undegraded":"2024-05-27T06:45:07.642764+0000","last_fullsized":"2024-05-27T06:45:07.642764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:04.541662+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:04.541662+0000","last_peered":"2024-05-27T06:45:04.541662+0000","last_clean":"2024-05-27T06:45:04.541662+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:04.541662+0000","last_undegraded":"2024-05-27T06:45:04.541662+0000","last_fullsized":"2024-05-27T06:45:04.541662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-27T06:45:33.611357+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:33.611357+0000","last_peered":"2024-05-27T06:45:33.611357+0000","last_clean":"2024-05-27T06:45:33.611357+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:33.611357+0000","last_undegraded":"2024-05-27T06:45:33.611357+0000","last_fullsized":"2024-05-27T06:45:33.611357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":668073,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":78007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":79537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":78041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":77067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":245760,"data_stored":77137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":311296,"data_stored":666847,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:47.443 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:47.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.633+0000 7f3b2ea00640 1 Processor -- start 2024-05-27T06:45:47.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.633+0000 7f3b2ea00640 1 -- start start 2024-05-27T06:45:47.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b280c6fd0 0x7f3b280c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ea00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 0x7f3b2815a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b2815db00 con 0x7f3b281546f0 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ea00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2815e6f0 con 0x7f3b280c6fd0 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ea00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f3b280755b0 con 0x7f3b2815a510 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b1fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b280c6fd0 0x7f3b280c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b27e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3b281546f0 0x7f3b28072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34276/0 (socket says 172.21.15.81:34276) 2024-05-27T06:45:47.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b27e00640 1 -- 172.21.15.81:0/668025030 learned_addr learned my addr 172.21.15.81:0/668025030 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b1fe00640 1 -- 172.21.15.81:0/668025030 >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 msgr2=0x7f3b2815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ca00640 1 --2- 172.21.15.81:0/668025030 >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 0x7f3b2815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b1fe00640 1 --2- 172.21.15.81:0/668025030 >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 0x7f3b2815a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b27400640 1 -- 172.21.15.81:0/668025030 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819003893 0 0) 0x7f3b2815db00 con 0x7f3b281546f0 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b1fe00640 1 -- 172.21.15.81:0/668025030 >> v1:172.21.15.81:6789/0 conn(0x7f3b281546f0 legacy=0x7f3b28072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b1fe00640 1 -- 172.21.15.81:0/668025030 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2815ae60 con 0x7f3b280c6fd0 2024-05-27T06:45:47.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.637+0000 7f3b2ca00640 1 --2- 172.21.15.81:0/668025030 >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 0x7f3b2815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:47.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b1fe00640 1 --2- 172.21.15.81:0/668025030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b280c6fd0 0x7f3b280c73e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3b280ca360 tx=0x7f3b1001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f33d1f489a41dd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b27400640 1 -- 172.21.15.81:0/668025030 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100b5070 con 0x7f3b280c6fd0 2024-05-27T06:45:47.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b27400640 1 -- 172.21.15.81:0/668025030 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b100a8970 con 0x7f3b280c6fd0 2024-05-27T06:45:47.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b27400640 1 -- 172.21.15.81:0/668025030 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100a8d40 con 0x7f3b280c6fd0 2024-05-27T06:45:47.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 -- 172.21.15.81:0/668025030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b280c6fd0 msgr2=0x7f3b280c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/668025030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b280c6fd0 0x7f3b280c73e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3b280ca360 tx=0x7f3b1001ec70 comp rx=0 tx=0).stop 2024-05-27T06:45:47.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 -- 172.21.15.81:0/668025030 shutdown_connections 2024-05-27T06:45:47.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/668025030 >> v2:172.21.15.106:3300/0 conn(0x7f3b2815a510 0x7f3b2815a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/668025030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b280c6fd0 0x7f3b280c73e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 -- 172.21.15.81:0/668025030 >> 172.21.15.81:0/668025030 conn(0x7f3b28070700 msgr2=0x7f3b280cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:47.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 -- 172.21.15.81:0/668025030 shutdown_connections 2024-05-27T06:45:47.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.641+0000 7f3b2ea00640 1 -- 172.21.15.81:0/668025030 wait complete. 2024-05-27T06:45:47.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 Processor -- start 2024-05-27T06:45:47.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- start start 2024-05-27T06:45:47.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 0x7f3b281f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2815a510 0x7f3b2815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2815e0a0 con 0x7f3b281546f0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2815ddc0 con 0x7f3b280c6fd0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2815e380 con 0x7f3b2815a510 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44240/0 (socket says 172.21.15.81:44240) 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 -- 172.21.15.81:0/3065767185 learned_addr learned my addr 172.21.15.81:0/3065767185 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b27e00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 0x7f3b281f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ca00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2815a510 0x7f3b2815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 -- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2815a510 msgr2=0x7f3b2815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2815a510 0x7f3b2815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 -- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 msgr2=0x7f3b281f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 0x7f3b281f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 -- 172.21.15.81:0/3065767185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2815c4b0 con 0x7f3b281546f0 2024-05-27T06:45:47.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b27e00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 0x7f3b281f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b1fe00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b100b3040 tx=0x7f3b1001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100b5040 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b100bf070 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100b0a80 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b281f6a20 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.645+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b281f6fe0 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2806a560 con 0x7f3b281546f0 2024-05-27T06:45:47.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3b100b0c20 con 0x7f3b281546f0 2024-05-27T06:45:47.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b24c00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 0x7f3b04071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:47.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3b1012d070 con 0x7f3b281546f0 2024-05-27T06:45:47.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b27e00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 0x7f3b04071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:47.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.649+0000 7f3b27e00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 0x7f3b04071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000f70 tx=0x7f3b18007cb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:47.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.653+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3b100c9020 con 0x7f3b281546f0 2024-05-27T06:45:47.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.809+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b2814e960 con 0x7f3b0406f430 2024-05-27T06:45:47.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.813+0000 7f3b24c00640 1 -- 172.21.15.81:0/3065767185 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252085 (secure 0 0 0) 0x7f3b18005c90 con 0x7f3b0406f430 2024-05-27T06:45:47.813 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:47.815 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:47.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 msgr2=0x7f3b04071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 0x7f3b04071940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b18000f70 tx=0x7f3b18007cb0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 msgr2=0x7f3b2815bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3b100b3040 tx=0x7f3b1001ba10 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 shutdown_connections 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b2815a510 0x7f3b2815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b0406f430 0x7f3b04071940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b281546f0 0x7f3b2815bf70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 --2- 172.21.15.81:0/3065767185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b280c6fd0 0x7f3b281f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 >> 172.21.15.81:0/3065767185 conn(0x7f3b28070700 msgr2=0x7f3b28159900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 shutdown_connections 2024-05-27T06:45:47.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:47.817+0000 7f3b2ea00640 1 -- 172.21.15.81:0/3065767185 wait complete. 2024-05-27T06:45:47.834 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-05-27T06:45:47.501484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600166,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":314,"num_write_kb":1698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218448,"kb_used_data":2260,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756272,"statfs":{"total":773094113280,"available":772870422528,"internally_reserved":0,"allocated":2314240,"data_stored":1896587,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1764,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899774+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:44.899774+0000","last_peered":"2024-05-27T06:45:44.899774+0000","last_clean":"2024-05-27T06:45:44.899774+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:44.899774+0000","last_undegraded":"2024-05-27T06:45:44.899774+0000","last_fullsized":"2024-05-27T06:45:44.899774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900482+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:44.900482+0000","last_peered":"2024-05-27T06:45:44.900482+0000","last_clean":"2024-05-27T06:45:44.900482+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:44.900482+0000","last_undegraded":"2024-05-27T06:45:44.900482+0000","last_fullsized":"2024-05-27T06:45:44.900482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626714+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:44.626714+0000","last_peered":"2024-05-27T06:45:44.626714+0000","last_clean":"2024-05-27T06:45:44.626714+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:44.626714+0000","last_undegraded":"2024-05-27T06:45:44.626714+0000","last_fullsized":"2024-05-27T06:45:44.626714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613831+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:44.613831+0000","last_peered":"2024-05-27T06:45:44.613831+0000","last_clean":"2024-05-27T06:45:44.613831+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:44.613831+0000","last_undegraded":"2024-05-27T06:45:44.613831+0000","last_fullsized":"2024-05-27T06:45:44.613831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899537+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:44.899537+0000","last_peered":"2024-05-27T06:45:44.899537+0000","last_clean":"2024-05-27T06:45:44.899537+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:44.899537+0000","last_undegraded":"2024-05-27T06:45:44.899537+0000","last_fullsized":"2024-05-27T06:45:44.899537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612724+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:44.612724+0000","last_peered":"2024-05-27T06:45:44.612724+0000","last_clean":"2024-05-27T06:45:44.612724+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:44.612724+0000","last_undegraded":"2024-05-27T06:45:44.612724+0000","last_fullsized":"2024-05-27T06:45:44.612724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899352+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:44.899352+0000","last_peered":"2024-05-27T06:45:44.899352+0000","last_clean":"2024-05-27T06:45:44.899352+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:44.899352+0000","last_undegraded":"2024-05-27T06:45:44.899352+0000","last_fullsized":"2024-05-27T06:45:44.899352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611373+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:44.611373+0000","last_peered":"2024-05-27T06:45:44.611373+0000","last_clean":"2024-05-27T06:45:44.611373+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:44.611373+0000","last_undegraded":"2024-05-27T06:45:44.611373+0000","last_fullsized":"2024-05-27T06:45:44.611373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899877+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:44.899877+0000","last_peered":"2024-05-27T06:45:44.899877+0000","last_clean":"2024-05-27T06:45:44.899877+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:44.899877+0000","last_undegraded":"2024-05-27T06:45:44.899877+0000","last_fullsized":"2024-05-27T06:45:44.899877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617211+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:44.617211+0000","last_peered":"2024-05-27T06:45:44.617211+0000","last_clean":"2024-05-27T06:45:44.617211+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:44.617211+0000","last_undegraded":"2024-05-27T06:45:44.617211+0000","last_fullsized":"2024-05-27T06:45:44.617211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.623110+0000","last_change":"2024-05-27T06:45:44.623110+0000","last_active":"2024-05-27T06:45:44.623110+0000","last_peered":"2024-05-27T06:45:44.623110+0000","last_clean":"2024-05-27T06:45:44.623110+0000","last_became_active":"2024-05-27T06:45:44.622703+0000","last_became_peered":"2024-05-27T06:45:44.622703+0000","last_unstale":"2024-05-27T06:45:44.623110+0000","last_undegraded":"2024-05-27T06:45:44.623110+0000","last_fullsized":"2024-05-27T06:45:44.623110+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.303150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614721+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:44.614721+0000","last_peered":"2024-05-27T06:45:44.614721+0000","last_clean":"2024-05-27T06:45:44.614721+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:44.614721+0000","last_undegraded":"2024-05-27T06:45:44.614721+0000","last_fullsized":"2024-05-27T06:45:44.614721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611625+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:44.611625+0000","last_peered":"2024-05-27T06:45:44.611625+0000","last_clean":"2024-05-27T06:45:44.611625+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:44.611625+0000","last_undegraded":"2024-05-27T06:45:44.611625+0000","last_fullsized":"2024-05-27T06:45:44.611625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626555+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:44.626555+0000","last_peered":"2024-05-27T06:45:44.626555+0000","last_clean":"2024-05-27T06:45:44.626555+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:44.626555+0000","last_undegraded":"2024-05-27T06:45:44.626555+0000","last_fullsized":"2024-05-27T06:45:44.626555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900159+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:44.900159+0000","last_peered":"2024-05-27T06:45:44.900159+0000","last_clean":"2024-05-27T06:45:44.900159+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:44.900159+0000","last_undegraded":"2024-05-27T06:45:44.900159+0000","last_fullsized":"2024-05-27T06:45:44.900159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613706+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:44.613706+0000","last_peered":"2024-05-27T06:45:44.613706+0000","last_clean":"2024-05-27T06:45:44.613706+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:44.613706+0000","last_undegraded":"2024-05-27T06:45:44.613706+0000","last_fullsized":"2024-05-27T06:45:44.613706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614231+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:44.614231+0000","last_peered":"2024-05-27T06:45:44.614231+0000","last_clean":"2024-05-27T06:45:44.614231+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:44.614231+0000","last_undegraded":"2024-05-27T06:45:44.614231+0000","last_fullsized":"2024-05-27T06:45:44.614231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615194+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:44.615194+0000","last_peered":"2024-05-27T06:45:44.615194+0000","last_clean":"2024-05-27T06:45:44.615194+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:44.615194+0000","last_undegraded":"2024-05-27T06:45:44.615194+0000","last_fullsized":"2024-05-27T06:45:44.615194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900043+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:44.900043+0000","last_peered":"2024-05-27T06:45:44.900043+0000","last_clean":"2024-05-27T06:45:44.900043+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:44.900043+0000","last_undegraded":"2024-05-27T06:45:44.900043+0000","last_fullsized":"2024-05-27T06:45:44.900043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617011+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:44.617011+0000","last_peered":"2024-05-27T06:45:44.617011+0000","last_clean":"2024-05-27T06:45:44.617011+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:44.617011+0000","last_undegraded":"2024-05-27T06:45:44.617011+0000","last_fullsized":"2024-05-27T06:45:44.617011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627177+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:44.627177+0000","last_peered":"2024-05-27T06:45:44.627177+0000","last_clean":"2024-05-27T06:45:44.627177+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:44.627177+0000","last_undegraded":"2024-05-27T06:45:44.627177+0000","last_fullsized":"2024-05-27T06:45:44.627177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612844+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:44.612844+0000","last_peered":"2024-05-27T06:45:44.612844+0000","last_clean":"2024-05-27T06:45:44.612844+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:44.612844+0000","last_undegraded":"2024-05-27T06:45:44.612844+0000","last_fullsized":"2024-05-27T06:45:44.612844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611421+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:44.611421+0000","last_peered":"2024-05-27T06:45:44.611421+0000","last_clean":"2024-05-27T06:45:44.611421+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:44.611421+0000","last_undegraded":"2024-05-27T06:45:44.611421+0000","last_fullsized":"2024-05-27T06:45:44.611421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613368+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:44.613368+0000","last_peered":"2024-05-27T06:45:44.613368+0000","last_clean":"2024-05-27T06:45:44.613368+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:44.613368+0000","last_undegraded":"2024-05-27T06:45:44.613368+0000","last_fullsized":"2024-05-27T06:45:44.613368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611891+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:44.611891+0000","last_peered":"2024-05-27T06:45:44.611891+0000","last_clean":"2024-05-27T06:45:44.611891+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:44.611891+0000","last_undegraded":"2024-05-27T06:45:44.611891+0000","last_fullsized":"2024-05-27T06:45:44.611891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617248+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:44.617248+0000","last_peered":"2024-05-27T06:45:44.617248+0000","last_clean":"2024-05-27T06:45:44.617248+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:44.617248+0000","last_undegraded":"2024-05-27T06:45:44.617248+0000","last_fullsized":"2024-05-27T06:45:44.617248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900102+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:44.900102+0000","last_peered":"2024-05-27T06:45:44.900102+0000","last_clean":"2024-05-27T06:45:44.900102+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:44.900102+0000","last_undegraded":"2024-05-27T06:45:44.900102+0000","last_fullsized":"2024-05-27T06:45:44.900102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612126+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:44.612126+0000","last_peered":"2024-05-27T06:45:44.612126+0000","last_clean":"2024-05-27T06:45:44.612126+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:44.612126+0000","last_undegraded":"2024-05-27T06:45:44.612126+0000","last_fullsized":"2024-05-27T06:45:44.612126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627290+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:44.627290+0000","last_peered":"2024-05-27T06:45:44.627290+0000","last_clean":"2024-05-27T06:45:44.627290+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:44.627290+0000","last_undegraded":"2024-05-27T06:45:44.627290+0000","last_fullsized":"2024-05-27T06:45:44.627290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612955+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:44.612955+0000","last_peered":"2024-05-27T06:45:44.612955+0000","last_clean":"2024-05-27T06:45:44.612955+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:44.612955+0000","last_undegraded":"2024-05-27T06:45:44.612955+0000","last_fullsized":"2024-05-27T06:45:44.612955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.902001+0000","last_change":"2024-05-27T06:45:44.902001+0000","last_active":"2024-05-27T06:45:44.902001+0000","last_peered":"2024-05-27T06:45:44.902001+0000","last_clean":"2024-05-27T06:45:44.902001+0000","last_became_active":"2024-05-27T06:45:44.901569+0000","last_became_peered":"2024-05-27T06:45:44.901569+0000","last_unstale":"2024-05-27T06:45:44.902001+0000","last_undegraded":"2024-05-27T06:45:44.902001+0000","last_fullsized":"2024-05-27T06:45:44.902001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:09.270195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616621+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:44.616621+0000","last_peered":"2024-05-27T06:45:44.616621+0000","last_clean":"2024-05-27T06:45:44.616621+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:44.616621+0000","last_undegraded":"2024-05-27T06:45:44.616621+0000","last_fullsized":"2024-05-27T06:45:44.616621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612493+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:44.612493+0000","last_peered":"2024-05-27T06:45:44.612493+0000","last_clean":"2024-05-27T06:45:44.612493+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:44.612493+0000","last_undegraded":"2024-05-27T06:45:44.612493+0000","last_fullsized":"2024-05-27T06:45:44.612493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612071+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:44.612071+0000","last_peered":"2024-05-27T06:45:44.612071+0000","last_clean":"2024-05-27T06:45:44.612071+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:44.612071+0000","last_undegraded":"2024-05-27T06:45:44.612071+0000","last_fullsized":"2024-05-27T06:45:44.612071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627388+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:44.627388+0000","last_peered":"2024-05-27T06:45:44.627388+0000","last_clean":"2024-05-27T06:45:44.627388+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:44.627388+0000","last_undegraded":"2024-05-27T06:45:44.627388+0000","last_fullsized":"2024-05-27T06:45:44.627388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616927+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:44.616927+0000","last_peered":"2024-05-27T06:45:44.616927+0000","last_clean":"2024-05-27T06:45:44.616927+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:44.616927+0000","last_undegraded":"2024-05-27T06:45:44.616927+0000","last_fullsized":"2024-05-27T06:45:44.616927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611614+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:44.611614+0000","last_peered":"2024-05-27T06:45:44.611614+0000","last_clean":"2024-05-27T06:45:44.611614+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:44.611614+0000","last_undegraded":"2024-05-27T06:45:44.611614+0000","last_fullsized":"2024-05-27T06:45:44.611614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611486+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:44.611486+0000","last_peered":"2024-05-27T06:45:44.611486+0000","last_clean":"2024-05-27T06:45:44.611486+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:44.611486+0000","last_undegraded":"2024-05-27T06:45:44.611486+0000","last_fullsized":"2024-05-27T06:45:44.611486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613522+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:44.613522+0000","last_peered":"2024-05-27T06:45:44.613522+0000","last_clean":"2024-05-27T06:45:44.613522+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:44.613522+0000","last_undegraded":"2024-05-27T06:45:44.613522+0000","last_fullsized":"2024-05-27T06:45:44.613522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899941+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:44.899941+0000","last_peered":"2024-05-27T06:45:44.899941+0000","last_clean":"2024-05-27T06:45:44.899941+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:44.899941+0000","last_undegraded":"2024-05-27T06:45:44.899941+0000","last_fullsized":"2024-05-27T06:45:44.899941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617354+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:44.617354+0000","last_peered":"2024-05-27T06:45:44.617354+0000","last_clean":"2024-05-27T06:45:44.617354+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:44.617354+0000","last_undegraded":"2024-05-27T06:45:44.617354+0000","last_fullsized":"2024-05-27T06:45:44.617354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614113+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:44.614113+0000","last_peered":"2024-05-27T06:45:44.614113+0000","last_clean":"2024-05-27T06:45:44.614113+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:44.614113+0000","last_undegraded":"2024-05-27T06:45:44.614113+0000","last_fullsized":"2024-05-27T06:45:44.614113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614231+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:44.614231+0000","last_peered":"2024-05-27T06:45:44.614231+0000","last_clean":"2024-05-27T06:45:44.614231+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:44.614231+0000","last_undegraded":"2024-05-27T06:45:44.614231+0000","last_fullsized":"2024-05-27T06:45:44.614231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626885+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:44.626885+0000","last_peered":"2024-05-27T06:45:44.626885+0000","last_clean":"2024-05-27T06:45:44.626885+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:44.626885+0000","last_undegraded":"2024-05-27T06:45:44.626885+0000","last_fullsized":"2024-05-27T06:45:44.626885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900253+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:44.900253+0000","last_peered":"2024-05-27T06:45:44.900253+0000","last_clean":"2024-05-27T06:45:44.900253+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:44.900253+0000","last_undegraded":"2024-05-27T06:45:44.900253+0000","last_fullsized":"2024-05-27T06:45:44.900253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614715+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:44.614715+0000","last_peered":"2024-05-27T06:45:44.614715+0000","last_clean":"2024-05-27T06:45:44.614715+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:44.614715+0000","last_undegraded":"2024-05-27T06:45:44.614715+0000","last_fullsized":"2024-05-27T06:45:44.614715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617081+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:44.617081+0000","last_peered":"2024-05-27T06:45:44.617081+0000","last_clean":"2024-05-27T06:45:44.617081+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:44.617081+0000","last_undegraded":"2024-05-27T06:45:44.617081+0000","last_fullsized":"2024-05-27T06:45:44.617081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899562+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:44.899562+0000","last_peered":"2024-05-27T06:45:44.899562+0000","last_clean":"2024-05-27T06:45:44.899562+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:44.899562+0000","last_undegraded":"2024-05-27T06:45:44.899562+0000","last_fullsized":"2024-05-27T06:45:44.899562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627479+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:44.627479+0000","last_peered":"2024-05-27T06:45:44.627479+0000","last_clean":"2024-05-27T06:45:44.627479+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:44.627479+0000","last_undegraded":"2024-05-27T06:45:44.627479+0000","last_fullsized":"2024-05-27T06:45:44.627479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626602+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:44.626602+0000","last_peered":"2024-05-27T06:45:44.626602+0000","last_clean":"2024-05-27T06:45:44.626602+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:44.626602+0000","last_undegraded":"2024-05-27T06:45:44.626602+0000","last_fullsized":"2024-05-27T06:45:44.626602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613062+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:44.613062+0000","last_peered":"2024-05-27T06:45:44.613062+0000","last_clean":"2024-05-27T06:45:44.613062+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:44.613062+0000","last_undegraded":"2024-05-27T06:45:44.613062+0000","last_fullsized":"2024-05-27T06:45:44.613062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899707+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:44.899707+0000","last_peered":"2024-05-27T06:45:44.899707+0000","last_clean":"2024-05-27T06:45:44.899707+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:44.899707+0000","last_undegraded":"2024-05-27T06:45:44.899707+0000","last_fullsized":"2024-05-27T06:45:44.899707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616834+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:44.616834+0000","last_peered":"2024-05-27T06:45:44.616834+0000","last_clean":"2024-05-27T06:45:44.616834+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:44.616834+0000","last_undegraded":"2024-05-27T06:45:44.616834+0000","last_fullsized":"2024-05-27T06:45:44.616834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616656+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:44.616656+0000","last_peered":"2024-05-27T06:45:44.616656+0000","last_clean":"2024-05-27T06:45:44.616656+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:44.616656+0000","last_undegraded":"2024-05-27T06:45:44.616656+0000","last_fullsized":"2024-05-27T06:45:44.616656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614422+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:44.614422+0000","last_peered":"2024-05-27T06:45:44.614422+0000","last_clean":"2024-05-27T06:45:44.614422+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:44.614422+0000","last_undegraded":"2024-05-27T06:45:44.614422+0000","last_fullsized":"2024-05-27T06:45:44.614422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899346+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:44.899346+0000","last_peered":"2024-05-27T06:45:44.899346+0000","last_clean":"2024-05-27T06:45:44.899346+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:44.899346+0000","last_undegraded":"2024-05-27T06:45:44.899346+0000","last_fullsized":"2024-05-27T06:45:44.899346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615004+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:44.615004+0000","last_peered":"2024-05-27T06:45:44.615004+0000","last_clean":"2024-05-27T06:45:44.615004+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:44.615004+0000","last_undegraded":"2024-05-27T06:45:44.615004+0000","last_fullsized":"2024-05-27T06:45:44.615004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900205+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:44.900205+0000","last_peered":"2024-05-27T06:45:44.900205+0000","last_clean":"2024-05-27T06:45:44.900205+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:44.900205+0000","last_undegraded":"2024-05-27T06:45:44.900205+0000","last_fullsized":"2024-05-27T06:45:44.900205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615315+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:44.615315+0000","last_peered":"2024-05-27T06:45:44.615315+0000","last_clean":"2024-05-27T06:45:44.615315+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:44.615315+0000","last_undegraded":"2024-05-27T06:45:44.615315+0000","last_fullsized":"2024-05-27T06:45:44.615315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899930+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:44.899930+0000","last_peered":"2024-05-27T06:45:44.899930+0000","last_clean":"2024-05-27T06:45:44.899930+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:44.899930+0000","last_undegraded":"2024-05-27T06:45:44.899930+0000","last_fullsized":"2024-05-27T06:45:44.899930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900366+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:44.900366+0000","last_peered":"2024-05-27T06:45:44.900366+0000","last_clean":"2024-05-27T06:45:44.900366+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:44.900366+0000","last_undegraded":"2024-05-27T06:45:44.900366+0000","last_fullsized":"2024-05-27T06:45:44.900366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613985+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:44.613985+0000","last_peered":"2024-05-27T06:45:44.613985+0000","last_clean":"2024-05-27T06:45:44.613985+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:44.613985+0000","last_undegraded":"2024-05-27T06:45:44.613985+0000","last_fullsized":"2024-05-27T06:45:44.613985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614610+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:44.614610+0000","last_peered":"2024-05-27T06:45:44.614610+0000","last_clean":"2024-05-27T06:45:44.614610+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:44.614610+0000","last_undegraded":"2024-05-27T06:45:44.614610+0000","last_fullsized":"2024-05-27T06:45:44.614610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.625321+0000","last_change":"2024-05-27T06:45:44.622177+0000","last_active":"2024-05-27T06:45:44.625321+0000","last_peered":"2024-05-27T06:45:44.625321+0000","last_clean":"2024-05-27T06:45:44.625321+0000","last_became_active":"2024-05-27T06:45:44.621763+0000","last_became_peered":"2024-05-27T06:45:44.621763+0000","last_unstale":"2024-05-27T06:45:44.625321+0000","last_undegraded":"2024-05-27T06:45:44.625321+0000","last_fullsized":"2024-05-27T06:45:44.625321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:42.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":114,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899664+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:44.899664+0000","last_peered":"2024-05-27T06:45:44.899664+0000","last_clean":"2024-05-27T06:45:44.899664+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:44.899664+0000","last_undegraded":"2024-05-27T06:45:44.899664+0000","last_fullsized":"2024-05-27T06:45:44.899664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613760+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:44.613760+0000","last_peered":"2024-05-27T06:45:44.613760+0000","last_clean":"2024-05-27T06:45:44.613760+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:44.613760+0000","last_undegraded":"2024-05-27T06:45:44.613760+0000","last_fullsized":"2024-05-27T06:45:44.613760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611703+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:44.611703+0000","last_peered":"2024-05-27T06:45:44.611703+0000","last_clean":"2024-05-27T06:45:44.611703+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:44.611703+0000","last_undegraded":"2024-05-27T06:45:44.611703+0000","last_fullsized":"2024-05-27T06:45:44.611703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899693+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:44.899693+0000","last_peered":"2024-05-27T06:45:44.899693+0000","last_clean":"2024-05-27T06:45:44.899693+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:44.899693+0000","last_undegraded":"2024-05-27T06:45:44.899693+0000","last_fullsized":"2024-05-27T06:45:44.899693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626525+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:44.626525+0000","last_peered":"2024-05-27T06:45:44.626525+0000","last_clean":"2024-05-27T06:45:44.626525+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:44.626525+0000","last_undegraded":"2024-05-27T06:45:44.626525+0000","last_fullsized":"2024-05-27T06:45:44.626525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614884+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:44.614884+0000","last_peered":"2024-05-27T06:45:44.614884+0000","last_clean":"2024-05-27T06:45:44.614884+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:44.614884+0000","last_undegraded":"2024-05-27T06:45:44.614884+0000","last_fullsized":"2024-05-27T06:45:44.614884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899941+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:44.899941+0000","last_peered":"2024-05-27T06:45:44.899941+0000","last_clean":"2024-05-27T06:45:44.899941+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:44.899941+0000","last_undegraded":"2024-05-27T06:45:44.899941+0000","last_fullsized":"2024-05-27T06:45:44.899941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613172+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:44.613172+0000","last_peered":"2024-05-27T06:45:44.613172+0000","last_clean":"2024-05-27T06:45:44.613172+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:44.613172+0000","last_undegraded":"2024-05-27T06:45:44.613172+0000","last_fullsized":"2024-05-27T06:45:44.613172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614041+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:44.614041+0000","last_peered":"2024-05-27T06:45:44.614041+0000","last_clean":"2024-05-27T06:45:44.614041+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:44.614041+0000","last_undegraded":"2024-05-27T06:45:44.614041+0000","last_fullsized":"2024-05-27T06:45:44.614041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900079+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:44.900079+0000","last_peered":"2024-05-27T06:45:44.900079+0000","last_clean":"2024-05-27T06:45:44.900079+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:44.900079+0000","last_undegraded":"2024-05-27T06:45:44.900079+0000","last_fullsized":"2024-05-27T06:45:44.900079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614478+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:44.614478+0000","last_peered":"2024-05-27T06:45:44.614478+0000","last_clean":"2024-05-27T06:45:44.614478+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:44.614478+0000","last_undegraded":"2024-05-27T06:45:44.614478+0000","last_fullsized":"2024-05-27T06:45:44.614478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899801+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:44.899801+0000","last_peered":"2024-05-27T06:45:44.899801+0000","last_clean":"2024-05-27T06:45:44.899801+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:44.899801+0000","last_undegraded":"2024-05-27T06:45:44.899801+0000","last_fullsized":"2024-05-27T06:45:44.899801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900009+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:44.900009+0000","last_peered":"2024-05-27T06:45:44.900009+0000","last_clean":"2024-05-27T06:45:44.900009+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:44.900009+0000","last_undegraded":"2024-05-27T06:45:44.900009+0000","last_fullsized":"2024-05-27T06:45:44.900009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900518+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:44.900518+0000","last_peered":"2024-05-27T06:45:44.900518+0000","last_clean":"2024-05-27T06:45:44.900518+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:44.900518+0000","last_undegraded":"2024-05-27T06:45:44.900518+0000","last_fullsized":"2024-05-27T06:45:44.900518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612001+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:44.612001+0000","last_peered":"2024-05-27T06:45:44.612001+0000","last_clean":"2024-05-27T06:45:44.612001+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:44.612001+0000","last_undegraded":"2024-05-27T06:45:44.612001+0000","last_fullsized":"2024-05-27T06:45:44.612001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617432+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:44.617432+0000","last_peered":"2024-05-27T06:45:44.617432+0000","last_clean":"2024-05-27T06:45:44.617432+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:44.617432+0000","last_undegraded":"2024-05-27T06:45:44.617432+0000","last_fullsized":"2024-05-27T06:45:44.617432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626897+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:44.626897+0000","last_peered":"2024-05-27T06:45:44.626897+0000","last_clean":"2024-05-27T06:45:44.626897+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:44.626897+0000","last_undegraded":"2024-05-27T06:45:44.626897+0000","last_fullsized":"2024-05-27T06:45:44.626897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899338+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:44.899338+0000","last_peered":"2024-05-27T06:45:44.899338+0000","last_clean":"2024-05-27T06:45:44.899338+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:44.899338+0000","last_undegraded":"2024-05-27T06:45:44.899338+0000","last_fullsized":"2024-05-27T06:45:44.899338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613168+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:44.613168+0000","last_peered":"2024-05-27T06:45:44.613168+0000","last_clean":"2024-05-27T06:45:44.613168+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:44.613168+0000","last_undegraded":"2024-05-27T06:45:44.613168+0000","last_fullsized":"2024-05-27T06:45:44.613168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900178+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:44.900178+0000","last_peered":"2024-05-27T06:45:44.900178+0000","last_clean":"2024-05-27T06:45:44.900178+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:44.900178+0000","last_undegraded":"2024-05-27T06:45:44.900178+0000","last_fullsized":"2024-05-27T06:45:44.900178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611542+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:44.611542+0000","last_peered":"2024-05-27T06:45:44.611542+0000","last_clean":"2024-05-27T06:45:44.611542+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:44.611542+0000","last_undegraded":"2024-05-27T06:45:44.611542+0000","last_fullsized":"2024-05-27T06:45:44.611542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900260+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:44.900260+0000","last_peered":"2024-05-27T06:45:44.900260+0000","last_clean":"2024-05-27T06:45:44.900260+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:44.900260+0000","last_undegraded":"2024-05-27T06:45:44.900260+0000","last_fullsized":"2024-05-27T06:45:44.900260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.629521+0000","last_change":"2024-05-27T06:45:44.628253+0000","last_active":"2024-05-27T06:45:44.629521+0000","last_peered":"2024-05-27T06:45:44.629521+0000","last_clean":"2024-05-27T06:45:44.629521+0000","last_became_active":"2024-05-27T06:45:44.627902+0000","last_became_peered":"2024-05-27T06:45:44.627902+0000","last_unstale":"2024-05-27T06:45:44.629521+0000","last_undegraded":"2024-05-27T06:45:44.629521+0000","last_fullsized":"2024-05-27T06:45:44.629521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:53:04.324850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899775+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:44.899775+0000","last_peered":"2024-05-27T06:45:44.899775+0000","last_clean":"2024-05-27T06:45:44.899775+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:44.899775+0000","last_undegraded":"2024-05-27T06:45:44.899775+0000","last_fullsized":"2024-05-27T06:45:44.899775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616802+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:44.616802+0000","last_peered":"2024-05-27T06:45:44.616802+0000","last_clean":"2024-05-27T06:45:44.616802+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:44.616802+0000","last_undegraded":"2024-05-27T06:45:44.616802+0000","last_fullsized":"2024-05-27T06:45:44.616802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614387+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:44.614387+0000","last_peered":"2024-05-27T06:45:44.614387+0000","last_clean":"2024-05-27T06:45:44.614387+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:44.614387+0000","last_undegraded":"2024-05-27T06:45:44.614387+0000","last_fullsized":"2024-05-27T06:45:44.614387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899822+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:44.899822+0000","last_peered":"2024-05-27T06:45:44.899822+0000","last_clean":"2024-05-27T06:45:44.899822+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:44.899822+0000","last_undegraded":"2024-05-27T06:45:44.899822+0000","last_fullsized":"2024-05-27T06:45:44.899822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613835+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:44.613835+0000","last_peered":"2024-05-27T06:45:44.613835+0000","last_clean":"2024-05-27T06:45:44.613835+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:44.613835+0000","last_undegraded":"2024-05-27T06:45:44.613835+0000","last_fullsized":"2024-05-27T06:45:44.613835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899628+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:44.899628+0000","last_peered":"2024-05-27T06:45:44.899628+0000","last_clean":"2024-05-27T06:45:44.899628+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:44.899628+0000","last_undegraded":"2024-05-27T06:45:44.899628+0000","last_fullsized":"2024-05-27T06:45:44.899628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626783+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:44.626783+0000","last_peered":"2024-05-27T06:45:44.626783+0000","last_clean":"2024-05-27T06:45:44.626783+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:44.626783+0000","last_undegraded":"2024-05-27T06:45:44.626783+0000","last_fullsized":"2024-05-27T06:45:44.626783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611834+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:44.611834+0000","last_peered":"2024-05-27T06:45:44.611834+0000","last_clean":"2024-05-27T06:45:44.611834+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:44.611834+0000","last_undegraded":"2024-05-27T06:45:44.611834+0000","last_fullsized":"2024-05-27T06:45:44.611834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611721+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:44.611721+0000","last_peered":"2024-05-27T06:45:44.611721+0000","last_clean":"2024-05-27T06:45:44.611721+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:44.611721+0000","last_undegraded":"2024-05-27T06:45:44.611721+0000","last_fullsized":"2024-05-27T06:45:44.611721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626940+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:44.626940+0000","last_peered":"2024-05-27T06:45:44.626940+0000","last_clean":"2024-05-27T06:45:44.626940+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:44.626940+0000","last_undegraded":"2024-05-27T06:45:44.626940+0000","last_fullsized":"2024-05-27T06:45:44.626940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9798,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":680696,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":91071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":92160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":89558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":91105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":89690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":89319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":672988,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:47.838 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-05-27T06:45:47.838 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T06:45:48.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.025+0000 7fb42ea00640 1 Processor -- start 2024-05-27T06:45:48.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.025+0000 7fb42ea00640 1 -- start start 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4280c9770 0x7fb4280cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ea00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb428160590 0x7fb42806e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4280d06f0 con 0x7fb42806e290 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ea00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb42815d5e0 con 0x7fb4280c9770 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ea00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb42815c9d0 con 0x7fb428160590 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4280c9770 0x7fb4280cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb41fe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb42806e290 0x7fb42815fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34278/0 (socket says 172.21.15.81:34278) 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb41fe00640 1 -- 172.21.15.81:0/1114448234 learned_addr learned my addr 172.21.15.81:0/1114448234 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb42ca00640 1 --2- 172.21.15.81:0/1114448234 >> v2:172.21.15.106:3300/0 conn(0x7fb428160590 0x7fb42806e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427e00640 1 -- 172.21.15.81:0/1114448234 >> v2:172.21.15.106:3300/0 conn(0x7fb428160590 msgr2=0x7fb42806e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427e00640 1 --2- 172.21.15.81:0/1114448234 >> v2:172.21.15.106:3300/0 conn(0x7fb428160590 0x7fb42806e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427e00640 1 -- 172.21.15.81:0/1114448234 >> v1:172.21.15.81:6789/0 conn(0x7fb42806e290 legacy=0x7fb42815fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427400640 1 -- 172.21.15.81:0/1114448234 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184269583 0 0) 0x7fb410001120 con 0x7fb42806e290 2024-05-27T06:45:48.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.029+0000 7fb427e00640 1 -- 172.21.15.81:0/1114448234 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42815ccb0 con 0x7fb4280c9770 2024-05-27T06:45:48.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb427e00640 1 --2- 172.21.15.81:0/1114448234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb4280cfdf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb428073e40 tx=0x7fb41801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf9ce5ac0d0b23a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb427400640 1 -- 172.21.15.81:0/1114448234 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4180b0070 con 0x7fb4280c9770 2024-05-27T06:45:48.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb427400640 1 -- 172.21.15.81:0/1114448234 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4180ab600 con 0x7fb4280c9770 2024-05-27T06:45:48.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb427400640 1 -- 172.21.15.81:0/1114448234 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4180ab9d0 con 0x7fb4280c9770 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 -- 172.21.15.81:0/1114448234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 msgr2=0x7fb4280cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1114448234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb4280cfdf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb428073e40 tx=0x7fb41801c3f0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 -- 172.21.15.81:0/1114448234 shutdown_connections 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1114448234 >> v2:172.21.15.106:3300/0 conn(0x7fb428160590 0x7fb42806e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1114448234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb4280cfdf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 -- 172.21.15.81:0/1114448234 >> 172.21.15.81:0/1114448234 conn(0x7fb428068ab0 msgr2=0x7fb428068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 -- 172.21.15.81:0/1114448234 shutdown_connections 2024-05-27T06:45:48.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 -- 172.21.15.81:0/1114448234 wait complete. 2024-05-27T06:45:48.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.033+0000 7fb42ea00640 1 Processor -- start 2024-05-27T06:45:48.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- start start 2024-05-27T06:45:48.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb42806e290 0x7fb4280cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb428160590 0x7fb4280ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4280d04b0 con 0x7fb4280c9770 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb4280d0330 con 0x7fb428160590 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb428169910 con 0x7fb42806e290 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44272/0 (socket says 172.21.15.81:44272) 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 -- 172.21.15.81:0/1721243039 learned_addr learned my addr 172.21.15.81:0/1721243039 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ca00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb428160590 0x7fb4280ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb427e00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb42806e290 0x7fb4280cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 -- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb42806e290 msgr2=0x7fb4280cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb42806e290 0x7fb4280cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 -- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb428160590 msgr2=0x7fb4280ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb428160590 0x7fb4280ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 -- 172.21.15.81:0/1721243039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42815e2e0 con 0x7fb4280c9770 2024-05-27T06:45:48.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb41fe00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c430 tx=0x7fb41000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4100074e0 con 0x7fb4280c9770 2024-05-27T06:45:48.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4100047f0 con 0x7fb4280c9770 2024-05-27T06:45:48.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410004be0 con 0x7fb4280c9770 2024-05-27T06:45:48.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb428150000 con 0x7fb4280c9770 2024-05-27T06:45:48.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4280cc010 con 0x7fb4280c9770 2024-05-27T06:45:48.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.037+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb42814fd50 con 0x7fb4280c9770 2024-05-27T06:45:48.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.041+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb410011070 con 0x7fb4280c9770 2024-05-27T06:45:48.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.041+0000 7fb424c00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 0x7fb404071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.041+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb41008efa0 con 0x7fb4280c9770 2024-05-27T06:45:48.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.041+0000 7fb427e00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 0x7fb404071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.041+0000 7fb427e00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 0x7fb404071910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4180ad000 tx=0x7fb418008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.045+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb4280d06f0 con 0x7fb4280c9770 2024-05-27T06:45:48.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.205+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 --> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb42806a560 con 0x7fb40406f400 2024-05-27T06:45:48.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.209+0000 7fb424c00640 1 -- 172.21.15.81:0/1721243039 <== mgr.4107 v2:172.21.15.81:6832/1352206918 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252085 (secure 0 0 0) 0x7fb42806a560 con 0x7fb40406f400 2024-05-27T06:45:48.208 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:48.210 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-05-27T06:45:48.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 msgr2=0x7fb404071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 0x7fb404071910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4180ad000 tx=0x7fb418008040 comp rx=0 tx=0).stop 2024-05-27T06:45:48.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 msgr2=0x7fb42814f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c430 tx=0x7fb41000c8f0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 shutdown_connections 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb40406f400 0x7fb404071910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb428160590 0x7fb4280ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4280c9770 0x7fb42814f6b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 --2- 172.21.15.81:0/1721243039 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb42806e290 0x7fb4280cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 >> 172.21.15.81:0/1721243039 conn(0x7fb428068ab0 msgr2=0x7fb428147170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 shutdown_connections 2024-05-27T06:45:48.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.213+0000 7fb42ea00640 1 -- 172.21.15.81:0/1721243039 wait complete. 2024-05-27T06:45:48.229 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-05-27T06:45:47.501484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600166,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":314,"num_write_kb":1698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":259,"ondisk_log_size":259,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218448,"kb_used_data":2260,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756272,"statfs":{"total":773094113280,"available":772870422528,"internally_reserved":0,"allocated":2314240,"data_stored":1896587,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1764,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899774+0000","last_change":"2024-05-27T06:45:02.645997+0000","last_active":"2024-05-27T06:45:44.899774+0000","last_peered":"2024-05-27T06:45:44.899774+0000","last_clean":"2024-05-27T06:45:44.899774+0000","last_became_active":"2024-05-27T06:45:02.645589+0000","last_became_peered":"2024-05-27T06:45:02.645589+0000","last_unstale":"2024-05-27T06:45:44.899774+0000","last_undegraded":"2024-05-27T06:45:44.899774+0000","last_fullsized":"2024-05-27T06:45:44.899774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:06:30.800333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900482+0000","last_change":"2024-05-27T06:45:02.632522+0000","last_active":"2024-05-27T06:45:44.900482+0000","last_peered":"2024-05-27T06:45:44.900482+0000","last_clean":"2024-05-27T06:45:44.900482+0000","last_became_active":"2024-05-27T06:45:02.632016+0000","last_became_peered":"2024-05-27T06:45:02.632016+0000","last_unstale":"2024-05-27T06:45:44.900482+0000","last_undegraded":"2024-05-27T06:45:44.900482+0000","last_fullsized":"2024-05-27T06:45:44.900482+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:57:49.443622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626714+0000","last_change":"2024-05-27T06:45:02.628577+0000","last_active":"2024-05-27T06:45:44.626714+0000","last_peered":"2024-05-27T06:45:44.626714+0000","last_clean":"2024-05-27T06:45:44.626714+0000","last_became_active":"2024-05-27T06:45:02.628020+0000","last_became_peered":"2024-05-27T06:45:02.628020+0000","last_unstale":"2024-05-27T06:45:44.626714+0000","last_undegraded":"2024-05-27T06:45:44.626714+0000","last_fullsized":"2024-05-27T06:45:44.626714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:42:53.228121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613831+0000","last_change":"2024-05-27T06:45:02.642079+0000","last_active":"2024-05-27T06:45:44.613831+0000","last_peered":"2024-05-27T06:45:44.613831+0000","last_clean":"2024-05-27T06:45:44.613831+0000","last_became_active":"2024-05-27T06:45:02.641181+0000","last_became_peered":"2024-05-27T06:45:02.641181+0000","last_unstale":"2024-05-27T06:45:44.613831+0000","last_undegraded":"2024-05-27T06:45:44.613831+0000","last_fullsized":"2024-05-27T06:45:44.613831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899537+0000","last_change":"2024-05-27T06:45:02.645017+0000","last_active":"2024-05-27T06:45:44.899537+0000","last_peered":"2024-05-27T06:45:44.899537+0000","last_clean":"2024-05-27T06:45:44.899537+0000","last_became_active":"2024-05-27T06:45:02.640515+0000","last_became_peered":"2024-05-27T06:45:02.640515+0000","last_unstale":"2024-05-27T06:45:44.899537+0000","last_undegraded":"2024-05-27T06:45:44.899537+0000","last_fullsized":"2024-05-27T06:45:44.899537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:18:13.125314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612724+0000","last_change":"2024-05-27T06:45:02.643799+0000","last_active":"2024-05-27T06:45:44.612724+0000","last_peered":"2024-05-27T06:45:44.612724+0000","last_clean":"2024-05-27T06:45:44.612724+0000","last_became_active":"2024-05-27T06:45:02.643473+0000","last_became_peered":"2024-05-27T06:45:02.643473+0000","last_unstale":"2024-05-27T06:45:44.612724+0000","last_undegraded":"2024-05-27T06:45:44.612724+0000","last_fullsized":"2024-05-27T06:45:44.612724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899352+0000","last_change":"2024-05-27T06:45:02.642988+0000","last_active":"2024-05-27T06:45:44.899352+0000","last_peered":"2024-05-27T06:45:44.899352+0000","last_clean":"2024-05-27T06:45:44.899352+0000","last_became_active":"2024-05-27T06:45:02.640442+0000","last_became_peered":"2024-05-27T06:45:02.640442+0000","last_unstale":"2024-05-27T06:45:44.899352+0000","last_undegraded":"2024-05-27T06:45:44.899352+0000","last_fullsized":"2024-05-27T06:45:44.899352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:41:01.175859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611373+0000","last_change":"2024-05-27T06:45:02.631200+0000","last_active":"2024-05-27T06:45:44.611373+0000","last_peered":"2024-05-27T06:45:44.611373+0000","last_clean":"2024-05-27T06:45:44.611373+0000","last_became_active":"2024-05-27T06:45:02.630850+0000","last_became_peered":"2024-05-27T06:45:02.630850+0000","last_unstale":"2024-05-27T06:45:44.611373+0000","last_undegraded":"2024-05-27T06:45:44.611373+0000","last_fullsized":"2024-05-27T06:45:44.611373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899877+0000","last_change":"2024-05-27T06:45:02.648713+0000","last_active":"2024-05-27T06:45:44.899877+0000","last_peered":"2024-05-27T06:45:44.899877+0000","last_clean":"2024-05-27T06:45:44.899877+0000","last_became_active":"2024-05-27T06:45:02.648363+0000","last_became_peered":"2024-05-27T06:45:02.648363+0000","last_unstale":"2024-05-27T06:45:44.899877+0000","last_undegraded":"2024-05-27T06:45:44.899877+0000","last_fullsized":"2024-05-27T06:45:44.899877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:58.059450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617211+0000","last_change":"2024-05-27T06:45:02.646405+0000","last_active":"2024-05-27T06:45:44.617211+0000","last_peered":"2024-05-27T06:45:44.617211+0000","last_clean":"2024-05-27T06:45:44.617211+0000","last_became_active":"2024-05-27T06:45:02.646099+0000","last_became_peered":"2024-05-27T06:45:02.646099+0000","last_unstale":"2024-05-27T06:45:44.617211+0000","last_undegraded":"2024-05-27T06:45:44.617211+0000","last_fullsized":"2024-05-27T06:45:44.617211+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:36.571478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.623110+0000","last_change":"2024-05-27T06:45:44.623110+0000","last_active":"2024-05-27T06:45:44.623110+0000","last_peered":"2024-05-27T06:45:44.623110+0000","last_clean":"2024-05-27T06:45:44.623110+0000","last_became_active":"2024-05-27T06:45:44.622703+0000","last_became_peered":"2024-05-27T06:45:44.622703+0000","last_unstale":"2024-05-27T06:45:44.623110+0000","last_undegraded":"2024-05-27T06:45:44.623110+0000","last_fullsized":"2024-05-27T06:45:44.623110+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:02.303150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614721+0000","last_change":"2024-05-27T06:45:02.637232+0000","last_active":"2024-05-27T06:45:44.614721+0000","last_peered":"2024-05-27T06:45:44.614721+0000","last_clean":"2024-05-27T06:45:44.614721+0000","last_became_active":"2024-05-27T06:45:02.635429+0000","last_became_peered":"2024-05-27T06:45:02.635429+0000","last_unstale":"2024-05-27T06:45:44.614721+0000","last_undegraded":"2024-05-27T06:45:44.614721+0000","last_fullsized":"2024-05-27T06:45:44.614721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:16:03.424749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611625+0000","last_change":"2024-05-27T06:45:02.639420+0000","last_active":"2024-05-27T06:45:44.611625+0000","last_peered":"2024-05-27T06:45:44.611625+0000","last_clean":"2024-05-27T06:45:44.611625+0000","last_became_active":"2024-05-27T06:45:02.639070+0000","last_became_peered":"2024-05-27T06:45:02.639070+0000","last_unstale":"2024-05-27T06:45:44.611625+0000","last_undegraded":"2024-05-27T06:45:44.611625+0000","last_fullsized":"2024-05-27T06:45:44.611625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626555+0000","last_change":"2024-05-27T06:45:02.631646+0000","last_active":"2024-05-27T06:45:44.626555+0000","last_peered":"2024-05-27T06:45:44.626555+0000","last_clean":"2024-05-27T06:45:44.626555+0000","last_became_active":"2024-05-27T06:45:02.631263+0000","last_became_peered":"2024-05-27T06:45:02.631263+0000","last_unstale":"2024-05-27T06:45:44.626555+0000","last_undegraded":"2024-05-27T06:45:44.626555+0000","last_fullsized":"2024-05-27T06:45:44.626555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:14:54.127872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900159+0000","last_change":"2024-05-27T06:45:02.623931+0000","last_active":"2024-05-27T06:45:44.900159+0000","last_peered":"2024-05-27T06:45:44.900159+0000","last_clean":"2024-05-27T06:45:44.900159+0000","last_became_active":"2024-05-27T06:45:02.622780+0000","last_became_peered":"2024-05-27T06:45:02.622780+0000","last_unstale":"2024-05-27T06:45:44.900159+0000","last_undegraded":"2024-05-27T06:45:44.900159+0000","last_fullsized":"2024-05-27T06:45:44.900159+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:59:35.464837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613706+0000","last_change":"2024-05-27T06:45:02.645335+0000","last_active":"2024-05-27T06:45:44.613706+0000","last_peered":"2024-05-27T06:45:44.613706+0000","last_clean":"2024-05-27T06:45:44.613706+0000","last_became_active":"2024-05-27T06:45:02.644287+0000","last_became_peered":"2024-05-27T06:45:02.644287+0000","last_unstale":"2024-05-27T06:45:44.613706+0000","last_undegraded":"2024-05-27T06:45:44.613706+0000","last_fullsized":"2024-05-27T06:45:44.613706+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:50:10.536540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614231+0000","last_change":"2024-05-27T06:45:02.639650+0000","last_active":"2024-05-27T06:45:44.614231+0000","last_peered":"2024-05-27T06:45:44.614231+0000","last_clean":"2024-05-27T06:45:44.614231+0000","last_became_active":"2024-05-27T06:45:02.638915+0000","last_became_peered":"2024-05-27T06:45:02.638915+0000","last_unstale":"2024-05-27T06:45:44.614231+0000","last_undegraded":"2024-05-27T06:45:44.614231+0000","last_fullsized":"2024-05-27T06:45:44.614231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:08:31.658204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615194+0000","last_change":"2024-05-27T06:45:02.638045+0000","last_active":"2024-05-27T06:45:44.615194+0000","last_peered":"2024-05-27T06:45:44.615194+0000","last_clean":"2024-05-27T06:45:44.615194+0000","last_became_active":"2024-05-27T06:45:02.637712+0000","last_became_peered":"2024-05-27T06:45:02.637712+0000","last_unstale":"2024-05-27T06:45:44.615194+0000","last_undegraded":"2024-05-27T06:45:44.615194+0000","last_fullsized":"2024-05-27T06:45:44.615194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:37:47.063839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900043+0000","last_change":"2024-05-27T06:45:02.643614+0000","last_active":"2024-05-27T06:45:44.900043+0000","last_peered":"2024-05-27T06:45:44.900043+0000","last_clean":"2024-05-27T06:45:44.900043+0000","last_became_active":"2024-05-27T06:45:02.643272+0000","last_became_peered":"2024-05-27T06:45:02.643272+0000","last_unstale":"2024-05-27T06:45:44.900043+0000","last_undegraded":"2024-05-27T06:45:44.900043+0000","last_fullsized":"2024-05-27T06:45:44.900043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:23.923338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617011+0000","last_change":"2024-05-27T06:45:02.651088+0000","last_active":"2024-05-27T06:45:44.617011+0000","last_peered":"2024-05-27T06:45:44.617011+0000","last_clean":"2024-05-27T06:45:44.617011+0000","last_became_active":"2024-05-27T06:45:02.650733+0000","last_became_peered":"2024-05-27T06:45:02.650733+0000","last_unstale":"2024-05-27T06:45:44.617011+0000","last_undegraded":"2024-05-27T06:45:44.617011+0000","last_fullsized":"2024-05-27T06:45:44.617011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:31:17.270429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627177+0000","last_change":"2024-05-27T06:45:02.633252+0000","last_active":"2024-05-27T06:45:44.627177+0000","last_peered":"2024-05-27T06:45:44.627177+0000","last_clean":"2024-05-27T06:45:44.627177+0000","last_became_active":"2024-05-27T06:45:02.632944+0000","last_became_peered":"2024-05-27T06:45:02.632944+0000","last_unstale":"2024-05-27T06:45:44.627177+0000","last_undegraded":"2024-05-27T06:45:44.627177+0000","last_fullsized":"2024-05-27T06:45:44.627177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:53:33.120699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612844+0000","last_change":"2024-05-27T06:45:02.638304+0000","last_active":"2024-05-27T06:45:44.612844+0000","last_peered":"2024-05-27T06:45:44.612844+0000","last_clean":"2024-05-27T06:45:44.612844+0000","last_became_active":"2024-05-27T06:45:02.637800+0000","last_became_peered":"2024-05-27T06:45:02.637800+0000","last_unstale":"2024-05-27T06:45:44.612844+0000","last_undegraded":"2024-05-27T06:45:44.612844+0000","last_fullsized":"2024-05-27T06:45:44.612844+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611421+0000","last_change":"2024-05-27T06:45:02.641372+0000","last_active":"2024-05-27T06:45:44.611421+0000","last_peered":"2024-05-27T06:45:44.611421+0000","last_clean":"2024-05-27T06:45:44.611421+0000","last_became_active":"2024-05-27T06:45:02.639541+0000","last_became_peered":"2024-05-27T06:45:02.639541+0000","last_unstale":"2024-05-27T06:45:44.611421+0000","last_undegraded":"2024-05-27T06:45:44.611421+0000","last_fullsized":"2024-05-27T06:45:44.611421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613368+0000","last_change":"2024-05-27T06:45:02.645598+0000","last_active":"2024-05-27T06:45:44.613368+0000","last_peered":"2024-05-27T06:45:44.613368+0000","last_clean":"2024-05-27T06:45:44.613368+0000","last_became_active":"2024-05-27T06:45:02.644835+0000","last_became_peered":"2024-05-27T06:45:02.644835+0000","last_unstale":"2024-05-27T06:45:44.613368+0000","last_undegraded":"2024-05-27T06:45:44.613368+0000","last_fullsized":"2024-05-27T06:45:44.613368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611891+0000","last_change":"2024-05-27T06:45:02.645178+0000","last_active":"2024-05-27T06:45:44.611891+0000","last_peered":"2024-05-27T06:45:44.611891+0000","last_clean":"2024-05-27T06:45:44.611891+0000","last_became_active":"2024-05-27T06:45:02.644783+0000","last_became_peered":"2024-05-27T06:45:02.644783+0000","last_unstale":"2024-05-27T06:45:44.611891+0000","last_undegraded":"2024-05-27T06:45:44.611891+0000","last_fullsized":"2024-05-27T06:45:44.611891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617248+0000","last_change":"2024-05-27T06:45:02.652037+0000","last_active":"2024-05-27T06:45:44.617248+0000","last_peered":"2024-05-27T06:45:44.617248+0000","last_clean":"2024-05-27T06:45:44.617248+0000","last_became_active":"2024-05-27T06:45:02.651648+0000","last_became_peered":"2024-05-27T06:45:02.651648+0000","last_unstale":"2024-05-27T06:45:44.617248+0000","last_undegraded":"2024-05-27T06:45:44.617248+0000","last_fullsized":"2024-05-27T06:45:44.617248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:36:52.455093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900102+0000","last_change":"2024-05-27T06:45:02.631253+0000","last_active":"2024-05-27T06:45:44.900102+0000","last_peered":"2024-05-27T06:45:44.900102+0000","last_clean":"2024-05-27T06:45:44.900102+0000","last_became_active":"2024-05-27T06:45:02.630961+0000","last_became_peered":"2024-05-27T06:45:02.630961+0000","last_unstale":"2024-05-27T06:45:44.900102+0000","last_undegraded":"2024-05-27T06:45:44.900102+0000","last_fullsized":"2024-05-27T06:45:44.900102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:52:43.022178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612126+0000","last_change":"2024-05-27T06:45:02.641857+0000","last_active":"2024-05-27T06:45:44.612126+0000","last_peered":"2024-05-27T06:45:44.612126+0000","last_clean":"2024-05-27T06:45:44.612126+0000","last_became_active":"2024-05-27T06:45:02.639601+0000","last_became_peered":"2024-05-27T06:45:02.639601+0000","last_unstale":"2024-05-27T06:45:44.612126+0000","last_undegraded":"2024-05-27T06:45:44.612126+0000","last_fullsized":"2024-05-27T06:45:44.612126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:00:09.530854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627290+0000","last_change":"2024-05-27T06:45:02.634235+0000","last_active":"2024-05-27T06:45:44.627290+0000","last_peered":"2024-05-27T06:45:44.627290+0000","last_clean":"2024-05-27T06:45:44.627290+0000","last_became_active":"2024-05-27T06:45:02.633685+0000","last_became_peered":"2024-05-27T06:45:02.633685+0000","last_unstale":"2024-05-27T06:45:44.627290+0000","last_undegraded":"2024-05-27T06:45:44.627290+0000","last_fullsized":"2024-05-27T06:45:44.627290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:59:09.586691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612955+0000","last_change":"2024-05-27T06:45:02.643854+0000","last_active":"2024-05-27T06:45:44.612955+0000","last_peered":"2024-05-27T06:45:44.612955+0000","last_clean":"2024-05-27T06:45:44.612955+0000","last_became_active":"2024-05-27T06:45:02.643491+0000","last_became_peered":"2024-05-27T06:45:02.643491+0000","last_unstale":"2024-05-27T06:45:44.612955+0000","last_undegraded":"2024-05-27T06:45:44.612955+0000","last_fullsized":"2024-05-27T06:45:44.612955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:44:01.490797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.902001+0000","last_change":"2024-05-27T06:45:44.902001+0000","last_active":"2024-05-27T06:45:44.902001+0000","last_peered":"2024-05-27T06:45:44.902001+0000","last_clean":"2024-05-27T06:45:44.902001+0000","last_became_active":"2024-05-27T06:45:44.901569+0000","last_became_peered":"2024-05-27T06:45:44.901569+0000","last_unstale":"2024-05-27T06:45:44.902001+0000","last_undegraded":"2024-05-27T06:45:44.902001+0000","last_fullsized":"2024-05-27T06:45:44.902001+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:10:09.270195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616621+0000","last_change":"2024-05-27T06:45:02.644635+0000","last_active":"2024-05-27T06:45:44.616621+0000","last_peered":"2024-05-27T06:45:44.616621+0000","last_clean":"2024-05-27T06:45:44.616621+0000","last_became_active":"2024-05-27T06:45:02.644333+0000","last_became_peered":"2024-05-27T06:45:02.644333+0000","last_unstale":"2024-05-27T06:45:44.616621+0000","last_undegraded":"2024-05-27T06:45:44.616621+0000","last_fullsized":"2024-05-27T06:45:44.616621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:57:56.901968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612493+0000","last_change":"2024-05-27T06:45:02.630313+0000","last_active":"2024-05-27T06:45:44.612493+0000","last_peered":"2024-05-27T06:45:44.612493+0000","last_clean":"2024-05-27T06:45:44.612493+0000","last_became_active":"2024-05-27T06:45:02.629969+0000","last_became_peered":"2024-05-27T06:45:02.629969+0000","last_unstale":"2024-05-27T06:45:44.612493+0000","last_undegraded":"2024-05-27T06:45:44.612493+0000","last_fullsized":"2024-05-27T06:45:44.612493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:14:25.148102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612071+0000","last_change":"2024-05-27T06:45:01.645594+0000","last_active":"2024-05-27T06:45:44.612071+0000","last_peered":"2024-05-27T06:45:44.612071+0000","last_clean":"2024-05-27T06:45:44.612071+0000","last_became_active":"2024-05-27T06:45:01.645347+0000","last_became_peered":"2024-05-27T06:45:01.645347+0000","last_unstale":"2024-05-27T06:45:44.612071+0000","last_undegraded":"2024-05-27T06:45:44.612071+0000","last_fullsized":"2024-05-27T06:45:44.612071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:45:23.529833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1063,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627388+0000","last_change":"2024-05-27T06:45:02.648686+0000","last_active":"2024-05-27T06:45:44.627388+0000","last_peered":"2024-05-27T06:45:44.627388+0000","last_clean":"2024-05-27T06:45:44.627388+0000","last_became_active":"2024-05-27T06:45:02.648334+0000","last_became_peered":"2024-05-27T06:45:02.648334+0000","last_unstale":"2024-05-27T06:45:44.627388+0000","last_undegraded":"2024-05-27T06:45:44.627388+0000","last_fullsized":"2024-05-27T06:45:44.627388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:57:12.972851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616927+0000","last_change":"2024-05-27T06:45:01.642072+0000","last_active":"2024-05-27T06:45:44.616927+0000","last_peered":"2024-05-27T06:45:44.616927+0000","last_clean":"2024-05-27T06:45:44.616927+0000","last_became_active":"2024-05-27T06:45:01.641639+0000","last_became_peered":"2024-05-27T06:45:01.641639+0000","last_unstale":"2024-05-27T06:45:44.616927+0000","last_undegraded":"2024-05-27T06:45:44.616927+0000","last_fullsized":"2024-05-27T06:45:44.616927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:05:20.140099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611614+0000","last_change":"2024-05-27T06:45:01.644140+0000","last_active":"2024-05-27T06:45:44.611614+0000","last_peered":"2024-05-27T06:45:44.611614+0000","last_clean":"2024-05-27T06:45:44.611614+0000","last_became_active":"2024-05-27T06:45:01.643765+0000","last_became_peered":"2024-05-27T06:45:01.643765+0000","last_unstale":"2024-05-27T06:45:44.611614+0000","last_undegraded":"2024-05-27T06:45:44.611614+0000","last_fullsized":"2024-05-27T06:45:44.611614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:01:37.024419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611486+0000","last_change":"2024-05-27T06:45:02.636609+0000","last_active":"2024-05-27T06:45:44.611486+0000","last_peered":"2024-05-27T06:45:44.611486+0000","last_clean":"2024-05-27T06:45:44.611486+0000","last_became_active":"2024-05-27T06:45:02.636144+0000","last_became_peered":"2024-05-27T06:45:02.636144+0000","last_unstale":"2024-05-27T06:45:44.611486+0000","last_undegraded":"2024-05-27T06:45:44.611486+0000","last_fullsized":"2024-05-27T06:45:44.611486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:13:14.670209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613522+0000","last_change":"2024-05-27T06:45:01.631166+0000","last_active":"2024-05-27T06:45:44.613522+0000","last_peered":"2024-05-27T06:45:44.613522+0000","last_clean":"2024-05-27T06:45:44.613522+0000","last_became_active":"2024-05-27T06:45:01.630747+0000","last_became_peered":"2024-05-27T06:45:01.630747+0000","last_unstale":"2024-05-27T06:45:44.613522+0000","last_undegraded":"2024-05-27T06:45:44.613522+0000","last_fullsized":"2024-05-27T06:45:44.613522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:18:00.861497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899941+0000","last_change":"2024-05-27T06:45:02.619792+0000","last_active":"2024-05-27T06:45:44.899941+0000","last_peered":"2024-05-27T06:45:44.899941+0000","last_clean":"2024-05-27T06:45:44.899941+0000","last_became_active":"2024-05-27T06:45:02.619471+0000","last_became_peered":"2024-05-27T06:45:02.619471+0000","last_unstale":"2024-05-27T06:45:44.899941+0000","last_undegraded":"2024-05-27T06:45:44.899941+0000","last_fullsized":"2024-05-27T06:45:44.899941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:31:40.002578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617354+0000","last_change":"2024-05-27T06:45:01.646981+0000","last_active":"2024-05-27T06:45:44.617354+0000","last_peered":"2024-05-27T06:45:44.617354+0000","last_clean":"2024-05-27T06:45:44.617354+0000","last_became_active":"2024-05-27T06:45:01.646600+0000","last_became_peered":"2024-05-27T06:45:01.646600+0000","last_unstale":"2024-05-27T06:45:44.617354+0000","last_undegraded":"2024-05-27T06:45:44.617354+0000","last_fullsized":"2024-05-27T06:45:44.617354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:12:46.884973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614113+0000","last_change":"2024-05-27T06:45:02.639863+0000","last_active":"2024-05-27T06:45:44.614113+0000","last_peered":"2024-05-27T06:45:44.614113+0000","last_clean":"2024-05-27T06:45:44.614113+0000","last_became_active":"2024-05-27T06:45:02.639035+0000","last_became_peered":"2024-05-27T06:45:02.639035+0000","last_unstale":"2024-05-27T06:45:44.614113+0000","last_undegraded":"2024-05-27T06:45:44.614113+0000","last_fullsized":"2024-05-27T06:45:44.614113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:54:46.418753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614231+0000","last_change":"2024-05-27T06:45:01.647417+0000","last_active":"2024-05-27T06:45:44.614231+0000","last_peered":"2024-05-27T06:45:44.614231+0000","last_clean":"2024-05-27T06:45:44.614231+0000","last_became_active":"2024-05-27T06:45:01.643619+0000","last_became_peered":"2024-05-27T06:45:01.643619+0000","last_unstale":"2024-05-27T06:45:44.614231+0000","last_undegraded":"2024-05-27T06:45:44.614231+0000","last_fullsized":"2024-05-27T06:45:44.614231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:10:59.135719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626885+0000","last_change":"2024-05-27T06:45:02.646203+0000","last_active":"2024-05-27T06:45:44.626885+0000","last_peered":"2024-05-27T06:45:44.626885+0000","last_clean":"2024-05-27T06:45:44.626885+0000","last_became_active":"2024-05-27T06:45:02.645692+0000","last_became_peered":"2024-05-27T06:45:02.645692+0000","last_unstale":"2024-05-27T06:45:44.626885+0000","last_undegraded":"2024-05-27T06:45:44.626885+0000","last_fullsized":"2024-05-27T06:45:44.626885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:16:32.394326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900253+0000","last_change":"2024-05-27T06:45:01.635804+0000","last_active":"2024-05-27T06:45:44.900253+0000","last_peered":"2024-05-27T06:45:44.900253+0000","last_clean":"2024-05-27T06:45:44.900253+0000","last_became_active":"2024-05-27T06:45:01.635424+0000","last_became_peered":"2024-05-27T06:45:01.635424+0000","last_unstale":"2024-05-27T06:45:44.900253+0000","last_undegraded":"2024-05-27T06:45:44.900253+0000","last_fullsized":"2024-05-27T06:45:44.900253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:04:50.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614715+0000","last_change":"2024-05-27T06:45:02.637605+0000","last_active":"2024-05-27T06:45:44.614715+0000","last_peered":"2024-05-27T06:45:44.614715+0000","last_clean":"2024-05-27T06:45:44.614715+0000","last_became_active":"2024-05-27T06:45:02.636785+0000","last_became_peered":"2024-05-27T06:45:02.636785+0000","last_unstale":"2024-05-27T06:45:44.614715+0000","last_undegraded":"2024-05-27T06:45:44.614715+0000","last_fullsized":"2024-05-27T06:45:44.614715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:38:13.829926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"22'12","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617081+0000","last_change":"2024-05-27T06:45:01.645639+0000","last_active":"2024-05-27T06:45:44.617081+0000","last_peered":"2024-05-27T06:45:44.617081+0000","last_clean":"2024-05-27T06:45:44.617081+0000","last_became_active":"2024-05-27T06:45:01.645265+0000","last_became_peered":"2024-05-27T06:45:01.645265+0000","last_unstale":"2024-05-27T06:45:44.617081+0000","last_undegraded":"2024-05-27T06:45:44.617081+0000","last_fullsized":"2024-05-27T06:45:44.617081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:13:24.524878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899562+0000","last_change":"2024-05-27T06:45:02.650484+0000","last_active":"2024-05-27T06:45:44.899562+0000","last_peered":"2024-05-27T06:45:44.899562+0000","last_clean":"2024-05-27T06:45:44.899562+0000","last_became_active":"2024-05-27T06:45:02.647295+0000","last_became_peered":"2024-05-27T06:45:02.647295+0000","last_unstale":"2024-05-27T06:45:44.899562+0000","last_undegraded":"2024-05-27T06:45:44.899562+0000","last_fullsized":"2024-05-27T06:45:44.899562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:06:54.797661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":27,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.627479+0000","last_change":"2024-05-27T06:45:01.629024+0000","last_active":"2024-05-27T06:45:44.627479+0000","last_peered":"2024-05-27T06:45:44.627479+0000","last_clean":"2024-05-27T06:45:44.627479+0000","last_became_active":"2024-05-27T06:45:01.628645+0000","last_became_peered":"2024-05-27T06:45:01.628645+0000","last_unstale":"2024-05-27T06:45:44.627479+0000","last_undegraded":"2024-05-27T06:45:44.627479+0000","last_fullsized":"2024-05-27T06:45:44.627479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:32:06.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626602+0000","last_change":"2024-05-27T06:45:02.636242+0000","last_active":"2024-05-27T06:45:44.626602+0000","last_peered":"2024-05-27T06:45:44.626602+0000","last_clean":"2024-05-27T06:45:44.626602+0000","last_became_active":"2024-05-27T06:45:02.635818+0000","last_became_peered":"2024-05-27T06:45:02.635818+0000","last_unstale":"2024-05-27T06:45:44.626602+0000","last_undegraded":"2024-05-27T06:45:44.626602+0000","last_fullsized":"2024-05-27T06:45:44.626602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:13:33.780973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613062+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:44.613062+0000","last_peered":"2024-05-27T06:45:44.613062+0000","last_clean":"2024-05-27T06:45:44.613062+0000","last_became_active":"2024-05-27T06:45:01.644312+0000","last_became_peered":"2024-05-27T06:45:01.644312+0000","last_unstale":"2024-05-27T06:45:44.613062+0000","last_undegraded":"2024-05-27T06:45:44.613062+0000","last_fullsized":"2024-05-27T06:45:44.613062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899707+0000","last_change":"2024-05-27T06:45:02.654961+0000","last_active":"2024-05-27T06:45:44.899707+0000","last_peered":"2024-05-27T06:45:44.899707+0000","last_clean":"2024-05-27T06:45:44.899707+0000","last_became_active":"2024-05-27T06:45:02.652457+0000","last_became_peered":"2024-05-27T06:45:02.652457+0000","last_unstale":"2024-05-27T06:45:44.899707+0000","last_undegraded":"2024-05-27T06:45:44.899707+0000","last_fullsized":"2024-05-27T06:45:44.899707+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:05:31.713526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616834+0000","last_change":"2024-05-27T06:45:01.642077+0000","last_active":"2024-05-27T06:45:44.616834+0000","last_peered":"2024-05-27T06:45:44.616834+0000","last_clean":"2024-05-27T06:45:44.616834+0000","last_became_active":"2024-05-27T06:45:01.641655+0000","last_became_peered":"2024-05-27T06:45:01.641655+0000","last_unstale":"2024-05-27T06:45:44.616834+0000","last_undegraded":"2024-05-27T06:45:44.616834+0000","last_fullsized":"2024-05-27T06:45:44.616834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:12:23.942286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1029,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616656+0000","last_change":"2024-05-27T06:45:02.636949+0000","last_active":"2024-05-27T06:45:44.616656+0000","last_peered":"2024-05-27T06:45:44.616656+0000","last_clean":"2024-05-27T06:45:44.616656+0000","last_became_active":"2024-05-27T06:45:02.636636+0000","last_became_peered":"2024-05-27T06:45:02.636636+0000","last_unstale":"2024-05-27T06:45:44.616656+0000","last_undegraded":"2024-05-27T06:45:44.616656+0000","last_fullsized":"2024-05-27T06:45:44.616656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:03:01.961987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614422+0000","last_change":"2024-05-27T06:45:01.635086+0000","last_active":"2024-05-27T06:45:44.614422+0000","last_peered":"2024-05-27T06:45:44.614422+0000","last_clean":"2024-05-27T06:45:44.614422+0000","last_became_active":"2024-05-27T06:45:01.634773+0000","last_became_peered":"2024-05-27T06:45:01.634773+0000","last_unstale":"2024-05-27T06:45:44.614422+0000","last_undegraded":"2024-05-27T06:45:44.614422+0000","last_fullsized":"2024-05-27T06:45:44.614422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:55:27.957586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899346+0000","last_change":"2024-05-27T06:45:02.648040+0000","last_active":"2024-05-27T06:45:44.899346+0000","last_peered":"2024-05-27T06:45:44.899346+0000","last_clean":"2024-05-27T06:45:44.899346+0000","last_became_active":"2024-05-27T06:45:02.647709+0000","last_became_peered":"2024-05-27T06:45:02.647709+0000","last_unstale":"2024-05-27T06:45:44.899346+0000","last_undegraded":"2024-05-27T06:45:44.899346+0000","last_fullsized":"2024-05-27T06:45:44.899346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:29:52.221186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615004+0000","last_change":"2024-05-27T06:45:01.625694+0000","last_active":"2024-05-27T06:45:44.615004+0000","last_peered":"2024-05-27T06:45:44.615004+0000","last_clean":"2024-05-27T06:45:44.615004+0000","last_became_active":"2024-05-27T06:45:01.625213+0000","last_became_peered":"2024-05-27T06:45:01.625213+0000","last_unstale":"2024-05-27T06:45:44.615004+0000","last_undegraded":"2024-05-27T06:45:44.615004+0000","last_fullsized":"2024-05-27T06:45:44.615004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:48:28.929453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900205+0000","last_change":"2024-05-27T06:45:02.650876+0000","last_active":"2024-05-27T06:45:44.900205+0000","last_peered":"2024-05-27T06:45:44.900205+0000","last_clean":"2024-05-27T06:45:44.900205+0000","last_became_active":"2024-05-27T06:45:02.650564+0000","last_became_peered":"2024-05-27T06:45:02.650564+0000","last_unstale":"2024-05-27T06:45:44.900205+0000","last_undegraded":"2024-05-27T06:45:44.900205+0000","last_fullsized":"2024-05-27T06:45:44.900205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:23:10.872752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.615315+0000","last_change":"2024-05-27T06:45:01.640578+0000","last_active":"2024-05-27T06:45:44.615315+0000","last_peered":"2024-05-27T06:45:44.615315+0000","last_clean":"2024-05-27T06:45:44.615315+0000","last_became_active":"2024-05-27T06:45:01.640232+0000","last_became_peered":"2024-05-27T06:45:01.640232+0000","last_unstale":"2024-05-27T06:45:44.615315+0000","last_undegraded":"2024-05-27T06:45:44.615315+0000","last_fullsized":"2024-05-27T06:45:44.615315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:55:17.309151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899930+0000","last_change":"2024-05-27T06:45:02.653220+0000","last_active":"2024-05-27T06:45:44.899930+0000","last_peered":"2024-05-27T06:45:44.899930+0000","last_clean":"2024-05-27T06:45:44.899930+0000","last_became_active":"2024-05-27T06:45:02.652845+0000","last_became_peered":"2024-05-27T06:45:02.652845+0000","last_unstale":"2024-05-27T06:45:44.899930+0000","last_undegraded":"2024-05-27T06:45:44.899930+0000","last_fullsized":"2024-05-27T06:45:44.899930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:50:27.365786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900366+0000","last_change":"2024-05-27T06:45:01.629747+0000","last_active":"2024-05-27T06:45:44.900366+0000","last_peered":"2024-05-27T06:45:44.900366+0000","last_clean":"2024-05-27T06:45:44.900366+0000","last_became_active":"2024-05-27T06:45:01.629445+0000","last_became_peered":"2024-05-27T06:45:01.629445+0000","last_unstale":"2024-05-27T06:45:44.900366+0000","last_undegraded":"2024-05-27T06:45:44.900366+0000","last_fullsized":"2024-05-27T06:45:44.900366+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:02:59.211709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613985+0000","last_change":"2024-05-27T06:45:02.642164+0000","last_active":"2024-05-27T06:45:44.613985+0000","last_peered":"2024-05-27T06:45:44.613985+0000","last_clean":"2024-05-27T06:45:44.613985+0000","last_became_active":"2024-05-27T06:45:02.641942+0000","last_became_peered":"2024-05-27T06:45:02.641942+0000","last_unstale":"2024-05-27T06:45:44.613985+0000","last_undegraded":"2024-05-27T06:45:44.613985+0000","last_fullsized":"2024-05-27T06:45:44.613985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614610+0000","last_change":"2024-05-27T06:45:01.635780+0000","last_active":"2024-05-27T06:45:44.614610+0000","last_peered":"2024-05-27T06:45:44.614610+0000","last_clean":"2024-05-27T06:45:44.614610+0000","last_became_active":"2024-05-27T06:45:01.634882+0000","last_became_peered":"2024-05-27T06:45:01.634882+0000","last_unstale":"2024-05-27T06:45:44.614610+0000","last_undegraded":"2024-05-27T06:45:44.614610+0000","last_fullsized":"2024-05-27T06:45:44.614610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T17:12:52.430828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1611,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.625321+0000","last_change":"2024-05-27T06:45:44.622177+0000","last_active":"2024-05-27T06:45:44.625321+0000","last_peered":"2024-05-27T06:45:44.625321+0000","last_clean":"2024-05-27T06:45:44.625321+0000","last_became_active":"2024-05-27T06:45:44.621763+0000","last_became_peered":"2024-05-27T06:45:44.621763+0000","last_unstale":"2024-05-27T06:45:44.625321+0000","last_undegraded":"2024-05-27T06:45:44.625321+0000","last_fullsized":"2024-05-27T06:45:44.625321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:51:42.561831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":114,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899664+0000","last_change":"2024-05-27T06:44:53.194110+0000","last_active":"2024-05-27T06:45:44.899664+0000","last_peered":"2024-05-27T06:45:44.899664+0000","last_clean":"2024-05-27T06:45:44.899664+0000","last_became_active":"2024-05-27T06:44:53.192916+0000","last_became_peered":"2024-05-27T06:44:53.192916+0000","last_unstale":"2024-05-27T06:45:44.899664+0000","last_undegraded":"2024-05-27T06:45:44.899664+0000","last_fullsized":"2024-05-27T06:45:44.899664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:44:52.140799+0000","last_clean_scrub_stamp":"2024-05-27T06:44:52.140799+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:18:32.851138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613760+0000","last_change":"2024-05-27T06:45:01.628119+0000","last_active":"2024-05-27T06:45:44.613760+0000","last_peered":"2024-05-27T06:45:44.613760+0000","last_clean":"2024-05-27T06:45:44.613760+0000","last_became_active":"2024-05-27T06:45:01.627757+0000","last_became_peered":"2024-05-27T06:45:01.627757+0000","last_unstale":"2024-05-27T06:45:44.613760+0000","last_undegraded":"2024-05-27T06:45:44.613760+0000","last_fullsized":"2024-05-27T06:45:44.613760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:49:14.059825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611703+0000","last_change":"2024-05-27T06:45:02.649475+0000","last_active":"2024-05-27T06:45:44.611703+0000","last_peered":"2024-05-27T06:45:44.611703+0000","last_clean":"2024-05-27T06:45:44.611703+0000","last_became_active":"2024-05-27T06:45:02.649240+0000","last_became_peered":"2024-05-27T06:45:02.649240+0000","last_unstale":"2024-05-27T06:45:44.611703+0000","last_undegraded":"2024-05-27T06:45:44.611703+0000","last_fullsized":"2024-05-27T06:45:44.611703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:37:11.644060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899693+0000","last_change":"2024-05-27T06:45:01.638640+0000","last_active":"2024-05-27T06:45:44.899693+0000","last_peered":"2024-05-27T06:45:44.899693+0000","last_clean":"2024-05-27T06:45:44.899693+0000","last_became_active":"2024-05-27T06:45:01.638317+0000","last_became_peered":"2024-05-27T06:45:01.638317+0000","last_unstale":"2024-05-27T06:45:44.899693+0000","last_undegraded":"2024-05-27T06:45:44.899693+0000","last_fullsized":"2024-05-27T06:45:44.899693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:27:02.196485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626525+0000","last_change":"2024-05-27T06:45:02.626517+0000","last_active":"2024-05-27T06:45:44.626525+0000","last_peered":"2024-05-27T06:45:44.626525+0000","last_clean":"2024-05-27T06:45:44.626525+0000","last_became_active":"2024-05-27T06:45:02.625793+0000","last_became_peered":"2024-05-27T06:45:02.625793+0000","last_unstale":"2024-05-27T06:45:44.626525+0000","last_undegraded":"2024-05-27T06:45:44.626525+0000","last_fullsized":"2024-05-27T06:45:44.626525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:11:47.394758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'6","reported_seq":24,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614884+0000","last_change":"2024-05-27T06:45:01.644338+0000","last_active":"2024-05-27T06:45:44.614884+0000","last_peered":"2024-05-27T06:45:44.614884+0000","last_clean":"2024-05-27T06:45:44.614884+0000","last_became_active":"2024-05-27T06:45:01.643978+0000","last_became_peered":"2024-05-27T06:45:01.643978+0000","last_unstale":"2024-05-27T06:45:44.614884+0000","last_undegraded":"2024-05-27T06:45:44.614884+0000","last_fullsized":"2024-05-27T06:45:44.614884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:16.894785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899941+0000","last_change":"2024-05-27T06:45:02.623133+0000","last_active":"2024-05-27T06:45:44.899941+0000","last_peered":"2024-05-27T06:45:44.899941+0000","last_clean":"2024-05-27T06:45:44.899941+0000","last_became_active":"2024-05-27T06:45:02.622823+0000","last_became_peered":"2024-05-27T06:45:02.622823+0000","last_unstale":"2024-05-27T06:45:44.899941+0000","last_undegraded":"2024-05-27T06:45:44.899941+0000","last_fullsized":"2024-05-27T06:45:44.899941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:53:48.131074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613172+0000","last_change":"2024-05-27T06:45:01.627734+0000","last_active":"2024-05-27T06:45:44.613172+0000","last_peered":"2024-05-27T06:45:44.613172+0000","last_clean":"2024-05-27T06:45:44.613172+0000","last_became_active":"2024-05-27T06:45:01.626976+0000","last_became_peered":"2024-05-27T06:45:01.626976+0000","last_unstale":"2024-05-27T06:45:44.613172+0000","last_undegraded":"2024-05-27T06:45:44.613172+0000","last_fullsized":"2024-05-27T06:45:44.613172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:58:25.409635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614041+0000","last_change":"2024-05-27T06:45:02.639791+0000","last_active":"2024-05-27T06:45:44.614041+0000","last_peered":"2024-05-27T06:45:44.614041+0000","last_clean":"2024-05-27T06:45:44.614041+0000","last_became_active":"2024-05-27T06:45:02.639464+0000","last_became_peered":"2024-05-27T06:45:02.639464+0000","last_unstale":"2024-05-27T06:45:44.614041+0000","last_undegraded":"2024-05-27T06:45:44.614041+0000","last_fullsized":"2024-05-27T06:45:44.614041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:37:13.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900079+0000","last_change":"2024-05-27T06:45:01.628017+0000","last_active":"2024-05-27T06:45:44.900079+0000","last_peered":"2024-05-27T06:45:44.900079+0000","last_clean":"2024-05-27T06:45:44.900079+0000","last_became_active":"2024-05-27T06:45:01.627700+0000","last_became_peered":"2024-05-27T06:45:01.627700+0000","last_unstale":"2024-05-27T06:45:44.900079+0000","last_undegraded":"2024-05-27T06:45:44.900079+0000","last_fullsized":"2024-05-27T06:45:44.900079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T18:41:44.809632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614478+0000","last_change":"2024-05-27T06:45:02.641396+0000","last_active":"2024-05-27T06:45:44.614478+0000","last_peered":"2024-05-27T06:45:44.614478+0000","last_clean":"2024-05-27T06:45:44.614478+0000","last_became_active":"2024-05-27T06:45:02.641068+0000","last_became_peered":"2024-05-27T06:45:02.641068+0000","last_unstale":"2024-05-27T06:45:44.614478+0000","last_undegraded":"2024-05-27T06:45:44.614478+0000","last_fullsized":"2024-05-27T06:45:44.614478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:37:51.547866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899801+0000","last_change":"2024-05-27T06:45:01.644942+0000","last_active":"2024-05-27T06:45:44.899801+0000","last_peered":"2024-05-27T06:45:44.899801+0000","last_clean":"2024-05-27T06:45:44.899801+0000","last_became_active":"2024-05-27T06:45:01.644620+0000","last_became_peered":"2024-05-27T06:45:01.644620+0000","last_unstale":"2024-05-27T06:45:44.899801+0000","last_undegraded":"2024-05-27T06:45:44.899801+0000","last_fullsized":"2024-05-27T06:45:44.899801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:48:37.456393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900009+0000","last_change":"2024-05-27T06:45:02.622610+0000","last_active":"2024-05-27T06:45:44.900009+0000","last_peered":"2024-05-27T06:45:44.900009+0000","last_clean":"2024-05-27T06:45:44.900009+0000","last_became_active":"2024-05-27T06:45:02.622283+0000","last_became_peered":"2024-05-27T06:45:02.622283+0000","last_unstale":"2024-05-27T06:45:44.900009+0000","last_undegraded":"2024-05-27T06:45:44.900009+0000","last_fullsized":"2024-05-27T06:45:44.900009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:10:25.486708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900518+0000","last_change":"2024-05-27T06:45:01.641913+0000","last_active":"2024-05-27T06:45:44.900518+0000","last_peered":"2024-05-27T06:45:44.900518+0000","last_clean":"2024-05-27T06:45:44.900518+0000","last_became_active":"2024-05-27T06:45:01.641610+0000","last_became_peered":"2024-05-27T06:45:01.641610+0000","last_unstale":"2024-05-27T06:45:44.900518+0000","last_undegraded":"2024-05-27T06:45:44.900518+0000","last_fullsized":"2024-05-27T06:45:44.900518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:35:26.916556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.612001+0000","last_change":"2024-05-27T06:45:02.639187+0000","last_active":"2024-05-27T06:45:44.612001+0000","last_peered":"2024-05-27T06:45:44.612001+0000","last_clean":"2024-05-27T06:45:44.612001+0000","last_became_active":"2024-05-27T06:45:02.638807+0000","last_became_peered":"2024-05-27T06:45:02.638807+0000","last_unstale":"2024-05-27T06:45:44.612001+0000","last_undegraded":"2024-05-27T06:45:44.612001+0000","last_fullsized":"2024-05-27T06:45:44.612001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:11:42.791954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.617432+0000","last_change":"2024-05-27T06:45:01.646141+0000","last_active":"2024-05-27T06:45:44.617432+0000","last_peered":"2024-05-27T06:45:44.617432+0000","last_clean":"2024-05-27T06:45:44.617432+0000","last_became_active":"2024-05-27T06:45:01.645908+0000","last_became_peered":"2024-05-27T06:45:01.645908+0000","last_unstale":"2024-05-27T06:45:44.617432+0000","last_undegraded":"2024-05-27T06:45:44.617432+0000","last_fullsized":"2024-05-27T06:45:44.617432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:10:27.042694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626897+0000","last_change":"2024-05-27T06:45:02.631651+0000","last_active":"2024-05-27T06:45:44.626897+0000","last_peered":"2024-05-27T06:45:44.626897+0000","last_clean":"2024-05-27T06:45:44.626897+0000","last_became_active":"2024-05-27T06:45:02.630561+0000","last_became_peered":"2024-05-27T06:45:02.630561+0000","last_unstale":"2024-05-27T06:45:44.626897+0000","last_undegraded":"2024-05-27T06:45:44.626897+0000","last_fullsized":"2024-05-27T06:45:44.626897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T12:59:42.628517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899338+0000","last_change":"2024-05-27T06:45:01.641063+0000","last_active":"2024-05-27T06:45:44.899338+0000","last_peered":"2024-05-27T06:45:44.899338+0000","last_clean":"2024-05-27T06:45:44.899338+0000","last_became_active":"2024-05-27T06:45:01.640726+0000","last_became_peered":"2024-05-27T06:45:01.640726+0000","last_unstale":"2024-05-27T06:45:44.899338+0000","last_undegraded":"2024-05-27T06:45:44.899338+0000","last_fullsized":"2024-05-27T06:45:44.899338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:49:23.334903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613168+0000","last_change":"2024-05-27T06:45:02.624347+0000","last_active":"2024-05-27T06:45:44.613168+0000","last_peered":"2024-05-27T06:45:44.613168+0000","last_clean":"2024-05-27T06:45:44.613168+0000","last_became_active":"2024-05-27T06:45:02.623856+0000","last_became_peered":"2024-05-27T06:45:02.623856+0000","last_unstale":"2024-05-27T06:45:44.613168+0000","last_undegraded":"2024-05-27T06:45:44.613168+0000","last_fullsized":"2024-05-27T06:45:44.613168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:15:32.292080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":25,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900178+0000","last_change":"2024-05-27T06:45:01.627391+0000","last_active":"2024-05-27T06:45:44.900178+0000","last_peered":"2024-05-27T06:45:44.900178+0000","last_clean":"2024-05-27T06:45:44.900178+0000","last_became_active":"2024-05-27T06:45:01.627054+0000","last_became_peered":"2024-05-27T06:45:01.627054+0000","last_unstale":"2024-05-27T06:45:44.900178+0000","last_undegraded":"2024-05-27T06:45:44.900178+0000","last_fullsized":"2024-05-27T06:45:44.900178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:48:38.238396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611542+0000","last_change":"2024-05-27T06:45:02.642303+0000","last_active":"2024-05-27T06:45:44.611542+0000","last_peered":"2024-05-27T06:45:44.611542+0000","last_clean":"2024-05-27T06:45:44.611542+0000","last_became_active":"2024-05-27T06:45:02.641391+0000","last_became_peered":"2024-05-27T06:45:02.641391+0000","last_unstale":"2024-05-27T06:45:44.611542+0000","last_undegraded":"2024-05-27T06:45:44.611542+0000","last_fullsized":"2024-05-27T06:45:44.611542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T10:48:24.684259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":21,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.900260+0000","last_change":"2024-05-27T06:45:01.645410+0000","last_active":"2024-05-27T06:45:44.900260+0000","last_peered":"2024-05-27T06:45:44.900260+0000","last_clean":"2024-05-27T06:45:44.900260+0000","last_became_active":"2024-05-27T06:45:01.645105+0000","last_became_peered":"2024-05-27T06:45:01.645105+0000","last_unstale":"2024-05-27T06:45:44.900260+0000","last_undegraded":"2024-05-27T06:45:44.900260+0000","last_fullsized":"2024-05-27T06:45:44.900260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T16:24:17.621511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.629521+0000","last_change":"2024-05-27T06:45:44.628253+0000","last_active":"2024-05-27T06:45:44.629521+0000","last_peered":"2024-05-27T06:45:44.629521+0000","last_clean":"2024-05-27T06:45:44.629521+0000","last_became_active":"2024-05-27T06:45:44.627902+0000","last_became_peered":"2024-05-27T06:45:44.627902+0000","last_unstale":"2024-05-27T06:45:44.629521+0000","last_undegraded":"2024-05-27T06:45:44.629521+0000","last_fullsized":"2024-05-27T06:45:44.629521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:53:04.324850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899775+0000","last_change":"2024-05-27T06:45:01.646370+0000","last_active":"2024-05-27T06:45:44.899775+0000","last_peered":"2024-05-27T06:45:44.899775+0000","last_clean":"2024-05-27T06:45:44.899775+0000","last_became_active":"2024-05-27T06:45:01.646022+0000","last_became_peered":"2024-05-27T06:45:01.646022+0000","last_unstale":"2024-05-27T06:45:44.899775+0000","last_undegraded":"2024-05-27T06:45:44.899775+0000","last_fullsized":"2024-05-27T06:45:44.899775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:06:06.392188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.616802+0000","last_change":"2024-05-27T06:45:02.647467+0000","last_active":"2024-05-27T06:45:44.616802+0000","last_peered":"2024-05-27T06:45:44.616802+0000","last_clean":"2024-05-27T06:45:44.616802+0000","last_became_active":"2024-05-27T06:45:02.647088+0000","last_became_peered":"2024-05-27T06:45:02.647088+0000","last_unstale":"2024-05-27T06:45:44.616802+0000","last_undegraded":"2024-05-27T06:45:44.616802+0000","last_fullsized":"2024-05-27T06:45:44.616802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T09:42:06.342207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.614387+0000","last_change":"2024-05-27T06:45:02.631093+0000","last_active":"2024-05-27T06:45:44.614387+0000","last_peered":"2024-05-27T06:45:44.614387+0000","last_clean":"2024-05-27T06:45:44.614387+0000","last_became_active":"2024-05-27T06:45:02.630745+0000","last_became_peered":"2024-05-27T06:45:02.630745+0000","last_unstale":"2024-05-27T06:45:44.614387+0000","last_undegraded":"2024-05-27T06:45:44.614387+0000","last_fullsized":"2024-05-27T06:45:44.614387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:33:11.511771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":23,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899822+0000","last_change":"2024-05-27T06:45:01.643318+0000","last_active":"2024-05-27T06:45:44.899822+0000","last_peered":"2024-05-27T06:45:44.899822+0000","last_clean":"2024-05-27T06:45:44.899822+0000","last_became_active":"2024-05-27T06:45:01.642988+0000","last_became_peered":"2024-05-27T06:45:01.642988+0000","last_unstale":"2024-05-27T06:45:44.899822+0000","last_undegraded":"2024-05-27T06:45:44.899822+0000","last_fullsized":"2024-05-27T06:45:44.899822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T08:42:43.110197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.613835+0000","last_change":"2024-05-27T06:45:02.644250+0000","last_active":"2024-05-27T06:45:44.613835+0000","last_peered":"2024-05-27T06:45:44.613835+0000","last_clean":"2024-05-27T06:45:44.613835+0000","last_became_active":"2024-05-27T06:45:02.643961+0000","last_became_peered":"2024-05-27T06:45:02.643961+0000","last_unstale":"2024-05-27T06:45:44.613835+0000","last_undegraded":"2024-05-27T06:45:44.613835+0000","last_fullsized":"2024-05-27T06:45:44.613835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T07:12:09.479556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'12","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.899628+0000","last_change":"2024-05-27T06:45:01.649244+0000","last_active":"2024-05-27T06:45:44.899628+0000","last_peered":"2024-05-27T06:45:44.899628+0000","last_clean":"2024-05-27T06:45:44.899628+0000","last_became_active":"2024-05-27T06:45:01.648945+0000","last_became_peered":"2024-05-27T06:45:01.648945+0000","last_unstale":"2024-05-27T06:45:44.899628+0000","last_undegraded":"2024-05-27T06:45:44.899628+0000","last_fullsized":"2024-05-27T06:45:44.899628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T11:23:00.097307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626783+0000","last_change":"2024-05-27T06:45:02.646122+0000","last_active":"2024-05-27T06:45:44.626783+0000","last_peered":"2024-05-27T06:45:44.626783+0000","last_clean":"2024-05-27T06:45:44.626783+0000","last_became_active":"2024-05-27T06:45:02.645891+0000","last_became_peered":"2024-05-27T06:45:02.645891+0000","last_unstale":"2024-05-27T06:45:44.626783+0000","last_undegraded":"2024-05-27T06:45:44.626783+0000","last_fullsized":"2024-05-27T06:45:44.626783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T15:42:36.372189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611834+0000","last_change":"2024-05-27T06:45:01.639694+0000","last_active":"2024-05-27T06:45:44.611834+0000","last_peered":"2024-05-27T06:45:44.611834+0000","last_clean":"2024-05-27T06:45:44.611834+0000","last_became_active":"2024-05-27T06:45:01.639368+0000","last_became_peered":"2024-05-27T06:45:01.639368+0000","last_unstale":"2024-05-27T06:45:44.611834+0000","last_undegraded":"2024-05-27T06:45:44.611834+0000","last_fullsized":"2024-05-27T06:45:44.611834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T14:50:25.029187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.611721+0000","last_change":"2024-05-27T06:45:02.638208+0000","last_active":"2024-05-27T06:45:44.611721+0000","last_peered":"2024-05-27T06:45:44.611721+0000","last_clean":"2024-05-27T06:45:44.611721+0000","last_became_active":"2024-05-27T06:45:02.637737+0000","last_became_peered":"2024-05-27T06:45:02.637737+0000","last_unstale":"2024-05-27T06:45:44.611721+0000","last_undegraded":"2024-05-27T06:45:44.611721+0000","last_fullsized":"2024-05-27T06:45:44.611721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:01.596452+0000","last_clean_scrub_stamp":"2024-05-27T06:45:01.596452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T13:03:00.455708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-27T06:45:44.626940+0000","last_change":"2024-05-27T06:45:01.635470+0000","last_active":"2024-05-27T06:45:44.626940+0000","last_peered":"2024-05-27T06:45:44.626940+0000","last_clean":"2024-05-27T06:45:44.626940+0000","last_became_active":"2024-05-27T06:45:01.635072+0000","last_became_peered":"2024-05-27T06:45:01.635072+0000","last_unstale":"2024-05-27T06:45:44.626940+0000","last_undegraded":"2024-05-27T06:45:44.626940+0000","last_fullsized":"2024-05-27T06:45:44.626940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-27T06:45:00.587126+0000","last_clean_scrub_stamp":"2024-05-27T06:45:00.587126+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-28T06:48:52.286090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9798,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":19596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":66596,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":344064,"data_stored":680696,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":91071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":92160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":89558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":91105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":89690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":89319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":672988,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":33298,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-27T06:45:48.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:45:48.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 Processor -- start 2024-05-27T06:45:48.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 -- start start 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f648806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f64880c8880 0x7f64880c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64880c9f80 con 0x7f648815cb80 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f64880c58b0 con 0x7f64880c8880 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.425+0000 7f648cc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f64880c83f0 con 0x7f648806c790 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f648815cb80 0x7f6488152cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34290/0 (socket says 172.21.15.81:34290) 2024-05-27T06:45:48.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 -- 172.21.15.81:0/3590231882 learned_addr learned my addr 172.21.15.81:0/3590231882 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:45:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6486800640 1 --2- 172.21.15.81:0/3590231882 >> v2:172.21.15.106:3300/0 conn(0x7f64880c8880 0x7f64880c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f647d400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f648806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288622295 0 0) 0x7f64880c9f80 con 0x7f648815cb80 2024-05-27T06:45:48.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f646c003530 con 0x7f648815cb80 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3722207479 0 0) 0x7f646c003530 con 0x7f648815cb80 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 msgr2=0x7f648806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f648806cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 >> v2:172.21.15.106:3300/0 conn(0x7f64880c8880 msgr2=0x7f64880c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 --2- 172.21.15.81:0/3590231882 >> v2:172.21.15.106:3300/0 conn(0x7f64880c8880 0x7f64880c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64880c52f0 con 0x7f648815cb80 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f6478001720 con 0x7f648815cb80 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 >> v1:172.21.15.81:6789/0 conn(0x7f648815cb80 legacy=0x7f6488152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f646c003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f646c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f646c004d50 0x7f646c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64880c58b0 con 0x7f646c003a10 2024-05-27T06:45:48.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f64880c83f0 con 0x7f646c004d50 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f646c00ba80 con 0x7f648806c790 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f646c003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f647d400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f646c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6486800640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f646c004d50 0x7f646c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 -- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 msgr2=0x7f646c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f646c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 -- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f646c004d50 msgr2=0x7f646c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f646c004d50 0x7f646c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f6485e00640 1 -- 172.21.15.81:0/3590231882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c003530 con 0x7f646c003a10 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.429+0000 7f647d400640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f648806c790 0x7f646c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:48.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f6485e00640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f646c003e80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f647800c290 tx=0x7f647801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92cc4e1cfb40eff server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478007d60 con 0x7f646c003a10 2024-05-27T06:45:48.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64780a9020 con 0x7f646c003a10 2024-05-27T06:45:48.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f6485400640 1 -- 172.21.15.81:0/3590231882 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6478005b00 con 0x7f646c003a10 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 -- 172.21.15.81:0/3590231882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 msgr2=0x7f646c003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 --2- 172.21.15.81:0/3590231882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f646c003e80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f647800c290 tx=0x7f647801cc60 comp rx=0 tx=0).stop 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f6486800640 1 -- 172.21.15.81:0/3590231882 reap_dead start 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 -- 172.21.15.81:0/3590231882 shutdown_connections 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 -- 172.21.15.81:0/3590231882 >> 172.21.15.81:0/3590231882 conn(0x7f6488070700 msgr2=0x7f64880cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 -- 172.21.15.81:0/3590231882 shutdown_connections 2024-05-27T06:45:48.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.433+0000 7f648cc00640 1 -- 172.21.15.81:0/3590231882 wait complete. 2024-05-27T06:45:48.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 Processor -- start 2024-05-27T06:45:48.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- start start 2024-05-27T06:45:48.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 0x7f648814ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 0x7f64881495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64880c9b00 con 0x7f646c003a10 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f648815cf90 con 0x7f64881501c0 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f64880c9c80 con 0x7f6488149190 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44286/0 (socket says 172.21.15.81:44286) 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 -- 172.21.15.81:0/1530860461 learned_addr learned my addr 172.21.15.81:0/1530860461 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 -- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 msgr2=0x7f64881495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6486800640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 0x7f64881495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f647d400640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 0x7f648814ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 0x7f64881495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 -- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 msgr2=0x7f648814ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 0x7f648814ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 -- 172.21.15.81:0/1530860461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6488151d70 con 0x7f646c003a10 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6486800640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 0x7f64881495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f647d400640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 0x7f648814ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:45:48.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f6485e00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f647801b360 tx=0x7f646c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64780049f0 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6478004b90 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6488150710 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64780070d0 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648815bd90 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.437+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648806a560 con 0x7f646c003a10 2024-05-27T06:45:48.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.441+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6478007270 con 0x7f646c003a10 2024-05-27T06:45:48.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.441+0000 7f647ea00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 0x7f6464071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.441+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6478128330 con 0x7f646c003a10 2024-05-27T06:45:48.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.441+0000 7f647d400640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 0x7f6464071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.441+0000 7f647d400640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 0x7f6464071910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f648814c3c0 tx=0x7f6470008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.445+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f64780b4070 con 0x7f646c003a10 2024-05-27T06:45:48.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.637+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f648815c640 con 0x7f646c003a10 2024-05-27T06:45:48.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f647ea00640 1 -- 172.21.15.81:0/1530860461 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f6478016a10 con 0x7f646c003a10 2024-05-27T06:45:48.636 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:48.636 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:45:48.636 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 msgr2=0x7f6464071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 0x7f6464071910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f648814c3c0 tx=0x7f6470008040 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 msgr2=0x7f6488151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f647801b360 tx=0x7f646c005840 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 shutdown_connections 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6488149190 0x7f64881495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f646406f400 0x7f6464071910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64881501c0 0x7f648814ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 --2- 172.21.15.81:0/1530860461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f646c003a10 0x7f6488151830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 >> 172.21.15.81:0/1530860461 conn(0x7f6488070700 msgr2=0x7f64880cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 shutdown_connections 2024-05-27T06:45:48.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.641+0000 7f648cc00640 1 -- 172.21.15.81:0/1530860461 wait complete. 2024-05-27T06:45:48.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-27T06:45:48.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 Processor -- start 2024-05-27T06:45:48.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 -- start start 2024-05-27T06:45:48.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6280c66a0 0x7fb6280ca260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6280cc420 con 0x7fb6280cb7e0 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb6280cc700 con 0x7fb6280c66a0 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb630400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb6280cc9e0 con 0x7fb62815a4c0 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb62da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb62d000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6280c66a0 0x7fb6280ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb62da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:55458/0 (socket says 172.21.15.81:55458) 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb62e400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb6280cb7e0 0x7fb628071f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:47088/0 (socket says 172.21.15.81:47088) 2024-05-27T06:45:48.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.809+0000 7fb62da00640 1 -- 172.21.15.81:0/2377935138 learned_addr learned my addr 172.21.15.81:0/2377935138 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:48.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62d000640 1 -- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 msgr2=0x7fb62815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:45:48.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62d000640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62d000640 1 -- 172.21.15.81:0/2377935138 >> v1:172.21.15.106:6789/0 conn(0x7fb6280cb7e0 legacy=0x7fb628071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62d000640 1 -- 172.21.15.81:0/2377935138 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62815dc50 con 0x7fb6280c66a0 2024-05-27T06:45:48.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62da00640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:45:48.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb62d000640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6280ca260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb61401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=861a3ab7afa334a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb627e00640 1 -- 172.21.15.81:0/2377935138 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b0070 con 0x7fb6280c66a0 2024-05-27T06:45:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb627e00640 1 -- 172.21.15.81:0/2377935138 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6140ab600 con 0x7fb6280c66a0 2024-05-27T06:45:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.813+0000 7fb627e00640 1 -- 172.21.15.81:0/2377935138 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140ab9d0 con 0x7fb6280c66a0 2024-05-27T06:45:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 msgr2=0x7fb6280ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6280ca260 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb6140097c0 tx=0x7fb61401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- 172.21.15.81:0/2377935138 shutdown_connections 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6280ca260 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- 172.21.15.81:0/2377935138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb62815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- 172.21.15.81:0/2377935138 >> 172.21.15.81:0/2377935138 conn(0x7fb628068cb0 msgr2=0x7fb6280745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- 172.21.15.81:0/2377935138 shutdown_connections 2024-05-27T06:45:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- 172.21.15.81:0/2377935138 wait complete. 2024-05-27T06:45:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 Processor -- start 2024-05-27T06:45:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- start start 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6281f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb628156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6280ca7a0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb6280ccff0 con 0x7fb6280c66a0 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.817+0000 7fb630400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb628167e60 con 0x7fb62815a4c0 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44290/0 (socket says 172.21.15.81:44290) 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 -- 172.21.15.81:0/1183966959 learned_addr learned my addr 172.21.15.81:0/1183966959 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62da00640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6281f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62e400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb628156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 -- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 msgr2=0x7fb628156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb628156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 -- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 msgr2=0x7fb6281f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6281f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 -- 172.21.15.81:0/1183966959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6281f46f0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb62d000640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb6140ae040 tx=0x7fb614019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b0040 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6140b43f0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6281573c0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140b4800 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6281576a0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62806a560 con 0x7fb6280cb7e0 2024-05-27T06:45:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.821+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb6140b49a0 con 0x7fb6280cb7e0 2024-05-27T06:45:48.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.825+0000 7fb625600640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 0x7fb608071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:45:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.825+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb614128d20 con 0x7fb6280cb7e0 2024-05-27T06:45:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.825+0000 7fb62da00640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 0x7fb608071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:45:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.825+0000 7fb62da00640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 0x7fb608071910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009a40 tx=0x7fb61c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:45:48.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.829+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb6140c2020 con 0x7fb6280cb7e0 2024-05-27T06:45:48.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.985+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb628075370 con 0x7fb6280cb7e0 2024-05-27T06:45:48.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb625600640 1 -- 172.21.15.81:0/1183966959 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16781 (secure 0 0 0) 0x7fb614016090 con 0x7fb6280cb7e0 2024-05-27T06:45:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:45:48.985 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"fsid":"cb957b26-d515-43f2-962a-6a97b6477eea","created":"2024-05-27T06:44:38.496166+0000","modified":"2024-05-27T06:45:44.606834+0000","last_up_change":"2024-05-27T06:44:50.135658+0000","last_in_change":"2024-05-27T06:44:41.140843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-27T06:44:51.516548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-27T06:45:00.358885+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-27T06:45:00.967125+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"990faba2-c472-4f9d-977d-aad42731b136","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6824","nonce":1838218899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6827","nonce":1838218899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6831","nonce":1838218899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1838218899},{"type":"v1","addr":"172.21.15.81:6829","nonce":1838218899}]},"public_addr":"172.21.15.81:6824/1838218899","cluster_addr":"172.21.15.81:6827/1838218899","heartbeat_back_addr":"172.21.15.81:6831/1838218899","heartbeat_front_addr":"172.21.15.81:6829/1838218899","state":["exists","up"]},{"osd":1,"uuid":"43a3497e-e0ac-484e-afb6-23dcb6fd48d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6801","nonce":404979354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6803","nonce":404979354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6807","nonce":404979354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":404979354},{"type":"v1","addr":"172.21.15.81:6805","nonce":404979354}]},"public_addr":"172.21.15.81:6801/404979354","cluster_addr":"172.21.15.81:6803/404979354","heartbeat_back_addr":"172.21.15.81:6807/404979354","heartbeat_front_addr":"172.21.15.81:6805/404979354","state":["exists","up"]},{"osd":2,"uuid":"e8bf3d65-09b9-490d-bbfc-1bc82b3f5ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6809","nonce":2305164086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6811","nonce":2305164086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6815","nonce":2305164086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":2305164086},{"type":"v1","addr":"172.21.15.81:6813","nonce":2305164086}]},"public_addr":"172.21.15.81:6809/2305164086","cluster_addr":"172.21.15.81:6811/2305164086","heartbeat_back_addr":"172.21.15.81:6815/2305164086","heartbeat_front_addr":"172.21.15.81:6813/2305164086","state":["exists","up"]},{"osd":3,"uuid":"b4f4f4e1-659d-4e0c-83d2-5ebeb5d57ce0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6817","nonce":2629536107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6819","nonce":2629536107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6825","nonce":2629536107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2629536107},{"type":"v1","addr":"172.21.15.81:6821","nonce":2629536107}]},"public_addr":"172.21.15.81:6817/2629536107","cluster_addr":"172.21.15.81:6819/2629536107","heartbeat_back_addr":"172.21.15.81:6825/2629536107","heartbeat_front_addr":"172.21.15.81:6821/2629536107","state":["exists","up"]},{"osd":4,"uuid":"5ccb2aa7-99d1-408f-8922-501785026e51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6802","nonce":735979092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6806","nonce":735979092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6814","nonce":735979092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":735979092},{"type":"v1","addr":"172.21.15.106:6810","nonce":735979092}]},"public_addr":"172.21.15.106:6802/735979092","cluster_addr":"172.21.15.106:6806/735979092","heartbeat_back_addr":"172.21.15.106:6814/735979092","heartbeat_front_addr":"172.21.15.106:6810/735979092","state":["exists","up"]},{"osd":5,"uuid":"74250015-c353-4668-a84c-2adf06bf7bf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6801","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6803","nonce":4003783298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6805","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6807","nonce":4003783298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6813","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6815","nonce":4003783298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6809","nonce":4003783298},{"type":"v1","addr":"172.21.15.106:6811","nonce":4003783298}]},"public_addr":"172.21.15.106:6803/4003783298","cluster_addr":"172.21.15.106:6807/4003783298","heartbeat_back_addr":"172.21.15.106:6815/4003783298","heartbeat_front_addr":"172.21.15.106:6811/4003783298","state":["exists","up"]},{"osd":6,"uuid":"afac5d3f-cbc1-4496-9f0f-833d2974d622","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6825","nonce":4186492770}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6827","nonce":4186492770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6831","nonce":4186492770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":4186492770},{"type":"v1","addr":"172.21.15.106:6829","nonce":4186492770}]},"public_addr":"172.21.15.106:6825/4186492770","cluster_addr":"172.21.15.106:6827/4186492770","heartbeat_back_addr":"172.21.15.106:6831/4186492770","heartbeat_front_addr":"172.21.15.106:6829/4186492770","state":["exists","up"]},{"osd":7,"uuid":"afafd7c5-606b-4724-acbc-f386c4f729b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6817","nonce":1738256587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6819","nonce":1738256587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6823","nonce":1738256587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":1738256587},{"type":"v1","addr":"172.21.15.106:6821","nonce":1738256587}]},"public_addr":"172.21.15.106:6817/1738256587","cluster_addr":"172.21.15.106:6819/1738256587","heartbeat_back_addr":"172.21.15.106:6823/1738256587","heartbeat_front_addr":"172.21.15.106:6821/1738256587","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.936622+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.082253+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.112951+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.869937+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.695488+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:47.091694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.964091+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-27T06:44:46.845686+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 msgr2=0x7fb608071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 0x7fb608071910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009a40 tx=0x7fb61c008040 comp rx=0 tx=0).stop 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 msgr2=0x7fb62814a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb6140ae040 tx=0x7fb614019240 comp rx=0 tx=0).stop 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 shutdown_connections 2024-05-27T06:45:48.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62815a4c0 0x7fb628156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60806f400 0x7fb608071910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6280cb7e0 0x7fb62814a0f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 --2- 172.21.15.81:0/1183966959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6280c66a0 0x7fb6281f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 >> 172.21.15.81:0/1183966959 conn(0x7fb628068cb0 msgr2=0x7fb628158dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 shutdown_connections 2024-05-27T06:45:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:45:48.989+0000 7fb630400640 1 -- 172.21.15.81:0/1183966959 wait complete. 2024-05-27T06:45:49.006 DEBUG:tasks.mds_thrash:Yielding 2024-05-27T06:45:49.007 INFO:teuthology.run_tasks:Running task workunit... 2024-05-27T06:45:49.023 INFO:tasks.workunit:Pulling workunits from ref 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:45:49.023 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-27T06:45:49.023 INFO:tasks.workunit:timeout=6h 2024-05-27T06:45:49.023 INFO:tasks.workunit:cleanup=True 2024-05-27T06:45:49.023 DEBUG:teuthology.orchestra.run.smithi081:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:49.024 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-05-27T06:45:49.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 81 secs before thrashing 2024-05-27T06:45:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:49.034 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 2ch/44d Inode: 1 Links: 2 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-05-27 06:45:07.629982254 +0000 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-05-27 06:45:45.269478945 +0000 2024-05-27T06:45:49.035 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2024-05-27T06:45:49.035 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-27T06:45:49.035 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-27T06:45:49.103 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 38f9d900c83c63799fbdbe61acc9a11b0d3554a6 2024-05-27T06:45:49.156 INFO:tasks.workunit.client.0.smithi081.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-27T06:46:01.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:46:01.559 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:46:31.619 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:46:31.623 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:47:01.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:47:01.695 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:47:10.025 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:10.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.267+0000 7f4fed200640 1 Processor -- start 2024-05-27T06:47:10.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.267+0000 7f4fed200640 1 -- start start 2024-05-27T06:47:10.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fed200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4fe80c6de0 0x7f4fe80cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fed200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 0x7f4fe80ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fed200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fe80755a0 con 0x7f4fe80cc930 2024-05-27T06:47:10.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fed200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4fe80c7400 con 0x7f4fe80c6de0 2024-05-27T06:47:10.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fed200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe80c98c0 con 0x7f4fe815a3e0 2024-05-27T06:47:10.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe6800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4fe80cc930 0x7f4fe80cf890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53360/0 (socket says 172.21.15.81:53360) 2024-05-27T06:47:10.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe6800640 1 -- 172.21.15.81:0/3262358657 learned_addr learned my addr 172.21.15.81:0/3262358657 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:10.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/3262358657 >> v2:172.21.15.106:3300/0 conn(0x7f4fe80c6de0 0x7f4fe80cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fdfe00640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 0x7f4fe80ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5400640 1 -- 172.21.15.81:0/3262358657 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432849474 0 0) 0x7f4fe80755a0 con 0x7f4fe80cc930 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5400640 1 -- 172.21.15.81:0/3262358657 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fcc003530 con 0x7f4fe80cc930 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5e00640 1 -- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 msgr2=0x7f4fe80ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 0x7f4fe80ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5e00640 1 -- 172.21.15.81:0/3262358657 >> v1:172.21.15.81:6789/0 conn(0x7f4fe80cc930 legacy=0x7f4fe80cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fe5e00640 1 -- 172.21.15.81:0/3262358657 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe80c9ba0 con 0x7f4fe80c6de0 2024-05-27T06:47:10.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.271+0000 7f4fdfe00640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 0x7f4fe80ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:47:10.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe80c6de0 0x7f4fe80cbd40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80097c0 tx=0x7f4fd801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df13dd07a5ee651 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fe5400640 1 -- 172.21.15.81:0/3262358657 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd80b5070 con 0x7f4fe80c6de0 2024-05-27T06:47:10.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fe5400640 1 -- 172.21.15.81:0/3262358657 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd80a8990 con 0x7f4fe80c6de0 2024-05-27T06:47:10.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fe5400640 1 -- 172.21.15.81:0/3262358657 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd80a8d60 con 0x7f4fe80c6de0 2024-05-27T06:47:10.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 -- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe80c6de0 msgr2=0x7f4fe80cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe80c6de0 0x7f4fe80cbd40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80097c0 tx=0x7f4fd801ec90 comp rx=0 tx=0).stop 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 -- 172.21.15.81:0/3262358657 shutdown_connections 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe815a3e0 0x7f4fe80ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 --2- 172.21.15.81:0/3262358657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe80c6de0 0x7f4fe80cbd40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 -- 172.21.15.81:0/3262358657 >> 172.21.15.81:0/3262358657 conn(0x7f4fe8068ab0 msgr2=0x7f4fe8068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 -- 172.21.15.81:0/3262358657 shutdown_connections 2024-05-27T06:47:10.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.275+0000 7f4fed200640 1 -- 172.21.15.81:0/3262358657 wait complete. 2024-05-27T06:47:10.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 Processor -- start 2024-05-27T06:47:10.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 -- start start 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 0x7f4fe815cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe815a3e0 0x7f4fe81e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe80c95c0 con 0x7f4fe80cc930 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe815a7f0 con 0x7f4fe815a3e0 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fed200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe80ca820 con 0x7f4fe80c6de0 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39274/0 (socket says 172.21.15.81:39274) 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 -- 172.21.15.81:0/1250238502 learned_addr learned my addr 172.21.15.81:0/1250238502 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fe6800640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe815a3e0 0x7f4fe81e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 0x7f4fe815cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 -- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 msgr2=0x7f4fe815cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 0x7f4fe815cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 -- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe815a3e0 msgr2=0x7f4fe81e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe815a3e0 0x7f4fe81e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 -- 172.21.15.81:0/1250238502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe8154240 con 0x7f4fe80cc930 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 0x7f4fe815cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fdfe00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000e9b0 tx=0x7f4fd000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.279+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0017070 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd000ca00 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000cdd0 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe80d0810 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe815b7f0 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fe806a560 con 0x7f4fe80cc930 2024-05-27T06:47:10.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4fd001a070 con 0x7f4fe80cc930 2024-05-27T06:47:10.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fde000640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 0x7f4fc4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.283+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4fd008f0d0 con 0x7f4fe80cc930 2024-05-27T06:47:10.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.287+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 0x7f4fc4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.287+0000 7f4fe5e00640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 0x7f4fc4071940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80b2000 tx=0x7f4fd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.291+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4fd005b970 con 0x7f4fe80cc930 2024-05-27T06:47:10.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.483+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fe80c90c0 con 0x7f4fe80cc930 2024-05-27T06:47:10.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.483+0000 7f4fde000640 1 -- 172.21.15.81:0/1250238502 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f4fd005fc70 con 0x7f4fe80cc930 2024-05-27T06:47:10.479 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:10.479 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:10.480 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:10.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 msgr2=0x7f4fc4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 0x7f4fc4071940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80b2000 tx=0x7f4fd8008040 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 msgr2=0x7f4fe81f2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000e9b0 tx=0x7f4fd000ee70 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 shutdown_connections 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fc406f430 0x7f4fc4071940 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fe815a3e0 0x7f4fe81e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fe80cc930 0x7f4fe81f2fd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 --2- 172.21.15.81:0/1250238502 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fe80c6de0 0x7f4fe815cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 >> 172.21.15.81:0/1250238502 conn(0x7f4fe8068ab0 msgr2=0x7f4fe8075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 shutdown_connections 2024-05-27T06:47:10.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.487+0000 7f4fed200640 1 -- 172.21.15.81:0/1250238502 wait complete. 2024-05-27T06:47:10.502 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2024-05-27T06:47:10.502 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-27T06:47:10.502 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:47:10.517 INFO:tasks.ceph.mds.d:Stopped 2024-05-27T06:47:10.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.679+0000 7f6b13000640 1 Processor -- start 2024-05-27T06:47:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.679+0000 7f6b13000640 1 -- start start 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b13000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b13000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6b0c165080 0x7f6b0c1674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b13000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0c160c20 con 0x7f6b0c0caca0 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b13000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0c06eb20 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b13000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0c06f910 con 0x7f6b0c165080 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39286/0 (socket says 172.21.15.81:39286) 2024-05-27T06:47:10.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 -- 172.21.15.81:0/2317191396 learned_addr learned my addr 172.21.15.81:0/2317191396 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b11000640 1 --2- 172.21.15.81:0/2317191396 >> v2:172.21.15.106:3300/0 conn(0x7f6b0c165080 0x7f6b0c1674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 -- 172.21.15.81:0/2317191396 >> v2:172.21.15.106:3300/0 conn(0x7f6b0c165080 msgr2=0x7f6b0c1674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 --2- 172.21.15.81:0/2317191396 >> v2:172.21.15.106:3300/0 conn(0x7f6b0c165080 0x7f6b0c1674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 -- 172.21.15.81:0/2317191396 >> v1:172.21.15.106:6789/0 conn(0x7f6b0c0caca0 legacy=0x7f6b0c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.683+0000 7f6b0be00640 1 -- 172.21.15.81:0/2317191396 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c15c710 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b0be00640 1 --2- 172.21.15.81:0/2317191396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0caa20 tx=0x7f6afc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6835c7e1a4d5ded2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b0b400640 1 -- 172.21.15.81:0/2317191396 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0a6040 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b0b400640 1 -- 172.21.15.81:0/2317191396 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6afc0af070 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b0b400640 1 -- 172.21.15.81:0/2317191396 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc0aa860 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 -- 172.21.15.81:0/2317191396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0ca2f0 msgr2=0x7f6b0c1606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 --2- 172.21.15.81:0/2317191396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c0caa20 tx=0x7f6afc01c520 comp rx=0 tx=0).stop 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 -- 172.21.15.81:0/2317191396 shutdown_connections 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 --2- 172.21.15.81:0/2317191396 >> v2:172.21.15.106:3300/0 conn(0x7f6b0c165080 0x7f6b0c1674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 --2- 172.21.15.81:0/2317191396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1606e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 -- 172.21.15.81:0/2317191396 >> 172.21.15.81:0/2317191396 conn(0x7f6b0c068ae0 msgr2=0x7f6b0c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 -- 172.21.15.81:0/2317191396 shutdown_connections 2024-05-27T06:47:10.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.687+0000 7f6b13000640 1 -- 172.21.15.81:0/2317191396 wait complete. 2024-05-27T06:47:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 Processor -- start 2024-05-27T06:47:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- start start 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b0c0caca0 0x7f6b0c166010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c15cf90 con 0x7f6b0c165080 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c15ccb0 con 0x7f6b0c0ca2f0 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0c06e220 con 0x7f6b0c0caca0 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39296/0 (socket says 172.21.15.81:39296) 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 -- 172.21.15.81:0/204704300 learned_addr learned my addr 172.21.15.81:0/204704300 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b03e00640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b0c0caca0 0x7f6b0c166010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b0be00640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 -- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b0c0caca0 msgr2=0x7f6b0c166010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b0c0caca0 0x7f6b0c166010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 -- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b0c0ca2f0 msgr2=0x7f6b0c1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 -- 172.21.15.81:0/204704300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c1fa380 con 0x7f6b0c165080 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b11000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b04007e60 tx=0x7f6b0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04014420 con 0x7f6b0c165080 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b04019070 con 0x7f6b0c165080 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c14f5b0 con 0x7f6b0c165080 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04014820 con 0x7f6b0c165080 2024-05-27T06:47:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.691+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0c165530 con 0x7f6b0c165080 2024-05-27T06:47:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.695+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b0c14f310 con 0x7f6b0c165080 2024-05-27T06:47:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.695+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6b040149c0 con 0x7f6b0c165080 2024-05-27T06:47:10.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.695+0000 7f6b08c00640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 0x7f6ae8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:10.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.695+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6b0408fa10 con 0x7f6b0c165080 2024-05-27T06:47:10.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.695+0000 7f6b0be00640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 0x7f6ae8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:10.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.699+0000 7f6b0be00640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 0x7f6ae8071910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0ae000 tx=0x7f6afc0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:10.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.699+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6b0c14f310 con 0x7f6b0c165080 2024-05-27T06:47:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.895+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b0c06a560 con 0x7f6b0c165080 2024-05-27T06:47:10.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.895+0000 7f6b08c00640 1 -- 172.21.15.81:0/204704300 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f6b0405c190 con 0x7f6b0c165080 2024-05-27T06:47:10.890 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:10.890 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:10.891 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:10.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 msgr2=0x7f6ae8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 0x7f6ae8071910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6afc0ae000 tx=0x7f6afc0a7000 comp rx=0 tx=0).stop 2024-05-27T06:47:10.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 msgr2=0x7f6b0c1f9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:10.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6b04007e60 tx=0x7f6b0400e4b0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 shutdown_connections 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6b0c0caca0 0x7f6b0c166010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ae806f400 0x7f6ae8071910 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b0c165080 0x7f6b0c1f9e40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 --2- 172.21.15.81:0/204704300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6b0c0ca2f0 0x7f6b0c1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 >> 172.21.15.81:0/204704300 conn(0x7f6b0c068ae0 msgr2=0x7f6b0c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 shutdown_connections 2024-05-27T06:47:10.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:10.899+0000 7f6b13000640 1 -- 172.21.15.81:0/204704300 wait complete. 2024-05-27T06:47:10.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:12.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:13.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 Processor -- start 2024-05-27T06:47:13.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 -- start start 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff3e40c6fd0 0x7ff3e40c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e415db00 con 0x7ff3e41546f0 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3e4073860 con 0x7ff3e415a510 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3ec400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff3e415e6f0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3e9a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff3e41546f0 0x7ff3e4072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53368/0 (socket says 172.21.15.81:53368) 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3e9a00640 1 -- 172.21.15.81:0/3117802645 learned_addr learned my addr 172.21.15.81:0/3117802645 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:13.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3e9000640 1 --2- 172.21.15.81:0/3117802645 >> v2:172.21.15.106:3300/0 conn(0x7ff3e40c6fd0 0x7ff3e40c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.070+0000 7ff3e3e00640 1 -- 172.21.15.81:0/3117802645 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685362583 0 0) 0x7ff3e415db00 con 0x7ff3e41546f0 2024-05-27T06:47:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e3e00640 1 -- 172.21.15.81:0/3117802645 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3cc003530 con 0x7ff3e41546f0 2024-05-27T06:47:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e9000640 1 -- 172.21.15.81:0/3117802645 >> v1:172.21.15.106:6789/0 conn(0x7ff3e415a510 legacy=0x7ff3e415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e9000640 1 -- 172.21.15.81:0/3117802645 >> v1:172.21.15.81:6789/0 conn(0x7ff3e41546f0 legacy=0x7ff3e4072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e9000640 1 -- 172.21.15.81:0/3117802645 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e415ddc0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e9000640 1 --2- 172.21.15.81:0/3117802645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e40c73e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00097c0 tx=0x7ff3d001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f17b500de7b9fe6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e3e00640 1 -- 172.21.15.81:0/3117802645 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d00b5070 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e3e00640 1 -- 172.21.15.81:0/3117802645 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d00a8990 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.074+0000 7ff3e3e00640 1 -- 172.21.15.81:0/3117802645 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d00a8d60 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- 172.21.15.81:0/3117802645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e40c6fd0 msgr2=0x7ff3e40c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3117802645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e40c73e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00097c0 tx=0x7ff3d001ec90 comp rx=0 tx=0).stop 2024-05-27T06:47:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- 172.21.15.81:0/3117802645 shutdown_connections 2024-05-27T06:47:13.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3117802645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e40c73e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- 172.21.15.81:0/3117802645 >> 172.21.15.81:0/3117802645 conn(0x7ff3e4070700 msgr2=0x7ff3e40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:13.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- 172.21.15.81:0/3117802645 shutdown_connections 2024-05-27T06:47:13.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- 172.21.15.81:0/3117802645 wait complete. 2024-05-27T06:47:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 Processor -- start 2024-05-27T06:47:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.078+0000 7ff3ec400640 1 -- start start 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3e41546f0 0x7ff3e415d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 0x7ff3e40c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e415e380 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e40755b0 con 0x7ff3e415a510 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e4154b00 con 0x7ff3e41546f0 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39298/0 (socket says 172.21.15.81:39298) 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 -- 172.21.15.81:0/3922410126 learned_addr learned my addr 172.21.15.81:0/3922410126 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9000640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3e41546f0 0x7ff3e415d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ea400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 0x7ff3e40c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 -- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3e41546f0 msgr2=0x7ff3e415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3e41546f0 0x7ff3e415d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 -- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 msgr2=0x7ff3e40c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 0x7ff3e40c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 -- 172.21.15.81:0/3922410126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e41f0fc0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ea400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 0x7ff3e40c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e9a00640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800c460 tx=0x7ff3d800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8017070 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d8005cf0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e41f1240 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8007270 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e41f1800 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.082+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e406a560 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.086+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff3d801a070 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.086+0000 7ff3e1600640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 0x7ff3c4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.086+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff3d808e550 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.086+0000 7ff3e9000640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 0x7ff3c4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.086+0000 7ff3e9000640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 0x7ff3c4071940 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00b2000 tx=0x7ff3d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:13.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.094+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff3e41f0fc0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.286+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3e40735a0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3e1600640 1 -- 172.21.15.81:0/3922410126 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7ff3d805acd0 con 0x7ff3e40c6fd0 2024-05-27T06:47:13.284 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:13.284 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:13.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 msgr2=0x7ff3c4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 0x7ff3c4071940 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff3d00b2000 tx=0x7ff3d0008040 comp rx=0 tx=0).stop 2024-05-27T06:47:13.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 msgr2=0x7ff3e4145d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:13.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800c460 tx=0x7ff3d800c920 comp rx=0 tx=0).stop 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 shutdown_connections 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3e41546f0 0x7ff3e415d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3c406f430 0x7ff3c4071940 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3e415a510 0x7ff3e40c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 --2- 172.21.15.81:0/3922410126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3e40c6fd0 0x7ff3e4145d30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 >> 172.21.15.81:0/3922410126 conn(0x7ff3e4070700 msgr2=0x7ff3e414a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 shutdown_connections 2024-05-27T06:47:13.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:13.290+0000 7ff3ec400640 1 -- 172.21.15.81:0/3922410126 wait complete. 2024-05-27T06:47:13.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:13.477 INFO:tasks.workunit.client.0.smithi081.stderr:Updating files: 67% (8235/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-27T06:47:14.289 INFO:tasks.workunit.client.0.smithi081.stderr:Note: switching to '38f9d900c83c63799fbdbe61acc9a11b0d3554a6'. 2024-05-27T06:47:14.289 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.289 INFO:tasks.workunit.client.0.smithi081.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-27T06:47:14.289 INFO:tasks.workunit.client.0.smithi081.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-27T06:47:14.289 INFO:tasks.workunit.client.0.smithi081.stderr:state without impacting any branches by switching back to a branch. 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: git switch -c 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr:Or undo this operation with: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: git switch - 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-05-27T06:47:14.290 INFO:tasks.workunit.client.0.smithi081.stderr:HEAD is now at 38f9d900c83 Merge PR #55758 into wip-vshankar-testing-20240521.063558-debug 2024-05-27T06:47:14.302 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-27T06:47:14.352 INFO:tasks.workunit.client.0.smithi081.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-27T06:47:14.354 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T06:47:14.355 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-27T06:47:15.118 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-27T06:47:15.201 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-27T06:47:15.285 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-27T06:47:15.287 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T06:47:15.287 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-27T06:47:15.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:15.341 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-27T06:47:15.348 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T06:47:15.348 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-27T06:47:15.404 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-05-27T06:47:15.406 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-05-27T06:47:15.406 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-05-27T06:47:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 Processor -- start 2024-05-27T06:47:15.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 -- start start 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00c9710 0x7f20f0072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20f00725d0 0x7f20f015fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f20f01603c0 0x7f20f0074630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f20f015d5a0 con 0x7f20f00725d0 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f20f01607d0 con 0x7f20f01603c0 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20f6c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00ca1e0 con 0x7f20f00c9710 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.462+0000 7f20e7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20f00725d0 0x7f20f015fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20f00725d0 0x7f20f015fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53240/0 (socket says 172.21.15.81:53240) 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 -- 172.21.15.81:0/4277765209 learned_addr learned my addr 172.21.15.81:0/4277765209 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20f4c00640 1 --2- 172.21.15.81:0/4277765209 >> v2:172.21.15.106:3300/0 conn(0x7f20f01603c0 0x7f20f0074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20efe00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00c9710 0x7f20f0072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 -- 172.21.15.81:0/4277765209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00c9710 msgr2=0x7f20f0072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00c9710 0x7f20f0072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 -- 172.21.15.81:0/4277765209 >> v2:172.21.15.106:3300/0 conn(0x7f20f01603c0 msgr2=0x7f20f0074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 --2- 172.21.15.81:0/4277765209 >> v2:172.21.15.106:3300/0 conn(0x7f20f01603c0 0x7f20f0074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 -- 172.21.15.81:0/4277765209 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00d0860 con 0x7f20f00725d0 2024-05-27T06:47:15.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20e7e00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f00725d0 0x7f20f015fe80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f20f0074480 tx=0x7f20d801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492ddd4b1f80aa57 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:15.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20ef400640 1 -- 172.21.15.81:0/4277765209 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80b0070 con 0x7f20f00725d0 2024-05-27T06:47:15.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20ef400640 1 -- 172.21.15.81:0/4277765209 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d80ab600 con 0x7f20f00725d0 2024-05-27T06:47:15.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.466+0000 7f20ef400640 1 -- 172.21.15.81:0/4277765209 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80ab9d0 con 0x7f20f00725d0 2024-05-27T06:47:15.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- 172.21.15.81:0/4277765209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f00725d0 msgr2=0x7f20f015fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:15.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f00725d0 0x7f20f015fe80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f20f0074480 tx=0x7f20d801c260 comp rx=0 tx=0).stop 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- 172.21.15.81:0/4277765209 shutdown_connections 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 --2- 172.21.15.81:0/4277765209 >> v2:172.21.15.106:3300/0 conn(0x7f20f01603c0 0x7f20f0074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f00725d0 0x7f20f015fe80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 --2- 172.21.15.81:0/4277765209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00c9710 0x7f20f0072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- 172.21.15.81:0/4277765209 >> 172.21.15.81:0/4277765209 conn(0x7f20f0068cb0 msgr2=0x7f20f00cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- 172.21.15.81:0/4277765209 shutdown_connections 2024-05-27T06:47:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- 172.21.15.81:0/4277765209 wait complete. 2024-05-27T06:47:15.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 Processor -- start 2024-05-27T06:47:15.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.470+0000 7f20f6c00640 1 -- start start 2024-05-27T06:47:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00725d0 0x7f20f01f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 0x7f20f0155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f20f015d230 con 0x7f20f01603c0 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f20f015cf50 con 0x7f20f00c9710 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00ca7b0 con 0x7f20f00725d0 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53256/0 (socket says 172.21.15.81:53256) 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 -- 172.21.15.81:0/3146268483 learned_addr learned my addr 172.21.15.81:0/3146268483 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20efe00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00725d0 0x7f20f01f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20e7e00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 0x7f20f0155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 -- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00725d0 msgr2=0x7f20f01f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00725d0 0x7f20f01f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 -- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 msgr2=0x7f20f0155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 0x7f20f0155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 -- 172.21.15.81:0/3146268483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00cb9e0 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20e7e00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 0x7f20f0155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f4c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f20e800efc0 tx=0x7f20e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8018020 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e8014480 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8009e60 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f00cbcc0 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f0159dc0 con 0x7f20f01603c0 2024-05-27T06:47:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.474+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f00c6810 con 0x7f20f01603c0 2024-05-27T06:47:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.478+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f20e801b020 con 0x7f20f01603c0 2024-05-27T06:47:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.478+0000 7f20ecc00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 0x7f20cc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.478+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f20e808fef0 con 0x7f20f01603c0 2024-05-27T06:47:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.478+0000 7f20efe00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 0x7f20cc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.478+0000 7f20efe00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 0x7f20cc071940 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009ac0 tx=0x7f20e0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.482+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f20e8022030 con 0x7f20f01603c0 2024-05-27T06:47:15.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20f006a560 con 0x7f20f01603c0 2024-05-27T06:47:15.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20ecc00640 1 -- 172.21.15.81:0/3146268483 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f20e8010040 con 0x7f20f01603c0 2024-05-27T06:47:15.673 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:15.673 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:15.674 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:15.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 msgr2=0x7f20cc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 0x7f20cc071940 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009ac0 tx=0x7f20e0009190 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 msgr2=0x7f20f014f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f20e800efc0 tx=0x7f20e800c490 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 shutdown_connections 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f20cc06f430 0x7f20cc071940 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20f01603c0 0x7f20f014f7b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20f00c9710 0x7f20f0155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 --2- 172.21.15.81:0/3146268483 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20f00725d0 0x7f20f01f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 >> 172.21.15.81:0/3146268483 conn(0x7f20f0068cb0 msgr2=0x7f20f015c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 shutdown_connections 2024-05-27T06:47:15.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:15.678+0000 7f20f6c00640 1 -- 172.21.15.81:0/3146268483 wait complete. 2024-05-27T06:47:15.696 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:17.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:17.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.858+0000 7f3f47e00640 1 Processor -- start 2024-05-27T06:47:17.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.858+0000 7f3f47e00640 1 -- start start 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f47e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3f400c5f70 0x7f3f40165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f47e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f40072840 con 0x7f3f400c5b60 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f47e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f400c5270 con 0x7f3f40072430 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f47e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f3f400c5040 con 0x7f3f400c5f70 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f44a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3f400c5b60 0x7f3f40161cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:58592/0 (socket says 172.21.15.81:58592) 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f44a00640 1 -- 172.21.15.81:0/832156254 learned_addr learned my addr 172.21.15.81:0/832156254 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:17.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f45e00640 1 --2- 172.21.15.81:0/832156254 >> v2:172.21.15.106:3300/0 conn(0x7f3f400c5f70 0x7f3f40165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 150681301 0 0) 0x7f3f40072840 con 0x7f3f400c5b60 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f28003530 con 0x7f3f400c5b60 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 796157713 0 0) 0x7f3f28003530 con 0x7f3f400c5b60 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f45e00640 1 -- 172.21.15.81:0/832156254 >> v1:172.21.15.106:6789/0 conn(0x7f3f40072430 legacy=0x7f3f40159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f45e00640 1 -- 172.21.15.81:0/832156254 >> v1:172.21.15.81:6789/0 conn(0x7f3f400c5b60 legacy=0x7f3f40161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:17.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.862+0000 7f3f45e00640 1 -- 172.21.15.81:0/832156254 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f400c7810 con 0x7f3f400c5f70 2024-05-27T06:47:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f45e00640 1 --2- 172.21.15.81:0/832156254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5f70 0x7f3f40165400 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009d90 tx=0x7f3f3801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba18d79a819b7db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c001700 con 0x7f3f400c5f70 2024-05-27T06:47:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f380a64e0 con 0x7f3f400c5f70 2024-05-27T06:47:17.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f3fe00640 1 -- 172.21.15.81:0/832156254 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380a6890 con 0x7f3f400c5f70 2024-05-27T06:47:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 -- 172.21.15.81:0/832156254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5f70 msgr2=0x7f3f40165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 --2- 172.21.15.81:0/832156254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5f70 0x7f3f40165400 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3f38009d90 tx=0x7f3f3801c890 comp rx=0 tx=0).stop 2024-05-27T06:47:17.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 -- 172.21.15.81:0/832156254 shutdown_connections 2024-05-27T06:47:17.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 --2- 172.21.15.81:0/832156254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5f70 0x7f3f40165400 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:17.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 -- 172.21.15.81:0/832156254 >> 172.21.15.81:0/832156254 conn(0x7f3f40070700 msgr2=0x7f3f400cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:17.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 -- 172.21.15.81:0/832156254 shutdown_connections 2024-05-27T06:47:17.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.866+0000 7f3f47e00640 1 -- 172.21.15.81:0/832156254 wait complete. 2024-05-27T06:47:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 Processor -- start 2024-05-27T06:47:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- start start 2024-05-27T06:47:17.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f40072430 0x7f3f401f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5b60 0x7f3f400cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f400c7dd0 con 0x7f3f400c5f70 2024-05-27T06:47:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3f400c75c0 con 0x7f3f400c5b60 2024-05-27T06:47:17.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3f400ce6f0 con 0x7f3f40072430 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53268/0 (socket says 172.21.15.81:53268) 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 -- 172.21.15.81:0/4051549979 learned_addr learned my addr 172.21.15.81:0/4051549979 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f44a00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5b60 0x7f3f400cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45400640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f40072430 0x7f3f401f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 -- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f40072430 msgr2=0x7f3f401f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f40072430 0x7f3f401f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 -- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5b60 msgr2=0x7f3f400cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5b60 0x7f3f400cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 -- 172.21.15.81:0/4051549979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4015a740 con 0x7f3f400c5f70 2024-05-27T06:47:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f45e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f380b0040 tx=0x7f3f38019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38009450 con 0x7f3f400c5f70 2024-05-27T06:47:17.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f380c1070 con 0x7f3f400c5f70 2024-05-27T06:47:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4015aa20 con 0x7f3f400c5f70 2024-05-27T06:47:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380b8aa0 con 0x7f3f400c5f70 2024-05-27T06:47:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.870+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f401561c0 con 0x7f3f400c5f70 2024-05-27T06:47:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4006a560 con 0x7f3f400c5f70 2024-05-27T06:47:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3f380b8c40 con 0x7f3f400c5f70 2024-05-27T06:47:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f3d600640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 0x7f3f20071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3f38129ff0 con 0x7f3f400c5f70 2024-05-27T06:47:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f45400640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 0x7f3f20071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:17.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.874+0000 7f3f45400640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 0x7f3f20071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f34007fd0 tx=0x7f3f34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:17.878+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3f38016480 con 0x7f3f400c5f70 2024-05-27T06:47:18.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.066+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f400c5940 con 0x7f3f400c5f70 2024-05-27T06:47:18.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f3d600640 1 -- 172.21.15.81:0/4051549979 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f3f380f67e0 con 0x7f3f400c5f70 2024-05-27T06:47:18.064 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:18.065 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:18.065 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 msgr2=0x7f3f20071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 0x7f3f20071910 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f34007fd0 tx=0x7f3f34009040 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 msgr2=0x7f3f4014d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3f380b0040 tx=0x7f3f38019630 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 shutdown_connections 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f2006f400 0x7f3f20071910 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f400c5f70 0x7f3f4014d180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f400c5b60 0x7f3f400cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 --2- 172.21.15.81:0/4051549979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f40072430 0x7f3f401f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:18.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 >> 172.21.15.81:0/4051549979 conn(0x7f3f40070700 msgr2=0x7f3f40075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 shutdown_connections 2024-05-27T06:47:18.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:18.070+0000 7f3f47e00640 1 -- 172.21.15.81:0/4051549979 wait complete. 2024-05-27T06:47:18.086 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:20.086 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:20.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 Processor -- start 2024-05-27T06:47:20.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 -- start start 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4f40ca2f0 0x7fb4f40caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f406e3e0 con 0x7fb4f40cfdc0 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f406e6c0 con 0x7fb4f415d300 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f9a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb4f415ce70 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4f40ca2f0 0x7fb4f40caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4f40ca2f0 0x7fb4f40caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53274/0 (socket says 172.21.15.81:53274) 2024-05-27T06:47:20.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f2a00640 1 -- 172.21.15.81:0/3012180879 learned_addr learned my addr 172.21.15.81:0/3012180879 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:20.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.250+0000 7fb4f3400640 1 --1- 172.21.15.81:0/3012180879 >> v1:172.21.15.81:6789/0 conn(0x7fb4f40cfdc0 0x7fb4f4167090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:58608/0 (socket says 172.21.15.81:58608) 2024-05-27T06:47:20.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2a00640 1 -- 172.21.15.81:0/3012180879 >> v1:172.21.15.106:6789/0 conn(0x7fb4f415d300 legacy=0x7fb4f4160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2a00640 1 -- 172.21.15.81:0/3012180879 >> v1:172.21.15.81:6789/0 conn(0x7fb4f40cfdc0 legacy=0x7fb4f4167090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2a00640 1 -- 172.21.15.81:0/3012180879 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f406e9a0 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2a00640 1 --2- 172.21.15.81:0/3012180879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f40caca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb4f415cff0 tx=0x7fb4e401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc6be171abde4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:20.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2000640 1 -- 172.21.15.81:0/3012180879 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40b1070 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2000640 1 -- 172.21.15.81:0/3012180879 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e40ac620 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f2000640 1 -- 172.21.15.81:0/3012180879 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e40ac9f0 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f9a00640 1 -- 172.21.15.81:0/3012180879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40ca2f0 msgr2=0x7fb4f40caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.254+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/3012180879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f40caca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb4f415cff0 tx=0x7fb4e401c4b0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- 172.21.15.81:0/3012180879 shutdown_connections 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/3012180879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f40caca0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- 172.21.15.81:0/3012180879 >> 172.21.15.81:0/3012180879 conn(0x7fb4f4068ae0 msgr2=0x7fb4f4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- 172.21.15.81:0/3012180879 shutdown_connections 2024-05-27T06:47:20.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- 172.21.15.81:0/3012180879 wait complete. 2024-05-27T06:47:20.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 Processor -- start 2024-05-27T06:47:20.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- start start 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f41fa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 0x7fb4f41f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f406f700 con 0x7fb4f415d300 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f415d710 con 0x7fb4f40ca2f0 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f9a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f406f880 con 0x7fb4f40cfdc0 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.258+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:20.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53276/0 (socket says 172.21.15.81:53276) 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 -- 172.21.15.81:0/686545014 learned_addr learned my addr 172.21.15.81:0/686545014 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f2a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f41fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4ea000640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 0x7fb4f41f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 -- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 msgr2=0x7fb4f41f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 0x7fb4f41f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 -- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4f40ca2f0 msgr2=0x7fb4f41fa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f41fa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 -- 172.21.15.81:0/686545014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f41f6e90 con 0x7fb4f415d300 2024-05-27T06:47:20.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4ea000640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 0x7fb4f41f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f3400640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00c700 tx=0x7fb4ec00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec017070 con 0x7fb4f415d300 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ec00ce10 con 0x7fb4f415d300 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec00a860 con 0x7fb4f415d300 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f4159930 con 0x7fb4f415d300 2024-05-27T06:47:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f4160a20 con 0x7fb4f415d300 2024-05-27T06:47:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f406a560 con 0x7fb4f415d300 2024-05-27T06:47:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.262+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb4ec01a070 con 0x7fb4f415d300 2024-05-27T06:47:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.266+0000 7fb4eb400640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 0x7fb4d0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.266+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb4ec08e900 con 0x7fb4f415d300 2024-05-27T06:47:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.266+0000 7fb4f2a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 0x7fb4d0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.266+0000 7fb4f2a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 0x7fb4d0071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40ae000 tx=0x7fb4e40a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:20.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.270+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb4ec05b080 con 0x7fb4f415d300 2024-05-27T06:47:20.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.458+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4f41f6600 con 0x7fb4f415d300 2024-05-27T06:47:20.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.458+0000 7fb4eb400640 1 -- 172.21.15.81:0/686545014 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7fb4ec05f380 con 0x7fb4f415d300 2024-05-27T06:47:20.454 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:20.454 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:20.454 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 msgr2=0x7fb4d0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 0x7fb4d0071940 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40ae000 tx=0x7fb4e40a7000 comp rx=0 tx=0).stop 2024-05-27T06:47:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 msgr2=0x7fb4f40cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00c700 tx=0x7fb4ec00cbc0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 shutdown_connections 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb4f40cfdc0 0x7fb4f41f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb4d006f430 0x7fb4d0071940 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f415d300 0x7fb4f40cc540 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 --2- 172.21.15.81:0/686545014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb4f40ca2f0 0x7fb4f41fa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 >> 172.21.15.81:0/686545014 conn(0x7fb4f4068ae0 msgr2=0x7fb4f40cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 shutdown_connections 2024-05-27T06:47:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:20.462+0000 7fb4f9a00640 1 -- 172.21.15.81:0/686545014 wait complete. 2024-05-27T06:47:20.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:22.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:22.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 Processor -- start 2024-05-27T06:47:22.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 -- start start 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 0x7f85f4159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f85f415a040 0x7f85f40c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f4072840 con 0x7f85f40c60a0 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f85f40c5040 con 0x7f85f415a040 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85fa200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f85f40c75c0 con 0x7f85f4072430 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85eaa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f85f415a040 0x7f85f40c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85eaa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f85f415a040 0x7f85f40c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53282/0 (socket says 172.21.15.81:53282) 2024-05-27T06:47:22.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.642+0000 7f85eaa00640 1 -- 172.21.15.81:0/522024966 learned_addr learned my addr 172.21.15.81:0/522024966 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85eaa00640 1 -- 172.21.15.81:0/522024966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 msgr2=0x7f85f4159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85f3400640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 0x7f85f4159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85eaa00640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 0x7f85f4159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85eaa00640 1 -- 172.21.15.81:0/522024966 >> v1:172.21.15.106:6789/0 conn(0x7f85f40c60a0 legacy=0x7f85f415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85eaa00640 1 -- 172.21.15.81:0/522024966 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f40c5270 con 0x7f85f415a040 2024-05-27T06:47:22.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85f3400640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 0x7f85f4159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:22.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85eaa00640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f40c5b60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009870 tx=0x7f85e401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102db67e9199ce50 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85f2a00640 1 -- 172.21.15.81:0/522024966 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e40b0070 con 0x7f85f415a040 2024-05-27T06:47:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85f2a00640 1 -- 172.21.15.81:0/522024966 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e40ab600 con 0x7f85f415a040 2024-05-27T06:47:22.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.646+0000 7f85f2a00640 1 -- 172.21.15.81:0/522024966 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e40ab9d0 con 0x7f85f415a040 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- 172.21.15.81:0/522024966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 msgr2=0x7f85f40c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f40c5b60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f85e4009870 tx=0x7f85e401c560 comp rx=0 tx=0).stop 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- 172.21.15.81:0/522024966 shutdown_connections 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f40c5b60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- 172.21.15.81:0/522024966 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f4072430 0x7f85f4159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- 172.21.15.81:0/522024966 >> 172.21.15.81:0/522024966 conn(0x7f85f4070700 msgr2=0x7f85f40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- 172.21.15.81:0/522024966 shutdown_connections 2024-05-27T06:47:22.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- 172.21.15.81:0/522024966 wait complete. 2024-05-27T06:47:22.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 Processor -- start 2024-05-27T06:47:22.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- start start 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f4072430 0x7f85f41409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f40c60a0 0x7f85f40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85f40ce6f0 con 0x7f85f415a040 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f85f40c7dd0 con 0x7f85f4072430 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.650+0000 7f85fa200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f85f40c5610 con 0x7f85f40c60a0 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53292/0 (socket says 172.21.15.81:53292) 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 -- 172.21.15.81:0/2254442845 learned_addr learned my addr 172.21.15.81:0/2254442845 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85eaa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f40c60a0 0x7f85f40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3400640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f4072430 0x7f85f41409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 -- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f40c60a0 msgr2=0x7f85f40ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f40c60a0 0x7f85f40ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 -- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f4072430 msgr2=0x7f85f41409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f4072430 0x7f85f41409a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 -- 172.21.15.81:0/2254442845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f4158cc0 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85f3e00640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00b9a0 tx=0x7f85ec00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec019070 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ec00cba0 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f4158fa0 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec014440 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f4140480 con 0x7f85f415a040 2024-05-27T06:47:22.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f41591c0 con 0x7f85f415a040 2024-05-27T06:47:22.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.654+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f85ec01c070 con 0x7f85f415a040 2024-05-27T06:47:22.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.658+0000 7f85ebe00640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 0x7f85d0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:22.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.658+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f85ec08ef10 con 0x7f85f415a040 2024-05-27T06:47:22.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.658+0000 7f85f3400640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 0x7f85d0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:22.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.658+0000 7f85f3400640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 0x7f85d0071940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85f4070fe0 tx=0x7f85dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.662+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f85ec05b720 con 0x7f85f415a040 2024-05-27T06:47:22.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.854+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85f406a560 con 0x7f85f415a040 2024-05-27T06:47:22.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.854+0000 7f85ebe00640 1 -- 172.21.15.81:0/2254442845 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f85ec05fa20 con 0x7f85f415a040 2024-05-27T06:47:22.850 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:22.851 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:22.851 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 msgr2=0x7f85d0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 0x7f85d0071940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85f4070fe0 tx=0x7f85dc008040 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 msgr2=0x7f85f413e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85ec00b9a0 tx=0x7f85ec00be60 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 shutdown_connections 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f85f40c60a0 0x7f85f40ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f85d006f430 0x7f85d0071940 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85f415a040 0x7f85f413e210 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 --2- 172.21.15.81:0/2254442845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f4072430 0x7f85f41409a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:22.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 >> 172.21.15.81:0/2254442845 conn(0x7f85f4070700 msgr2=0x7f85f4074bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:22.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 shutdown_connections 2024-05-27T06:47:22.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:22.858+0000 7f85fa200640 1 -- 172.21.15.81:0/2254442845 wait complete. 2024-05-27T06:47:22.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:24.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:25.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 Processor -- start 2024-05-27T06:47:25.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 -- start start 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16800ca770 0x7f16800cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1680154200 0x7f1680154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16800c78b0 con 0x7f16800cbd00 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1680154b50 con 0x7f16800ca770 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f1686600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16800c6ee0 con 0x7f1680154200 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f167f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16800ca770 0x7f16800cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f167f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16800ca770 0x7f16800cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53300/0 (socket says 172.21.15.81:53300) 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.038+0000 7f167f400640 1 -- 172.21.15.81:0/3245731949 learned_addr learned my addr 172.21.15.81:0/3245731949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167fe00640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1680154200 0x7f1680154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167f400640 1 -- 172.21.15.81:0/3245731949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1680154200 msgr2=0x7f1680154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:25.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167f400640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1680154200 0x7f1680154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167f400640 1 -- 172.21.15.81:0/3245731949 >> v1:172.21.15.106:6789/0 conn(0x7f16800cbd00 legacy=0x7f16800cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167f400640 1 -- 172.21.15.81:0/3245731949 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16800c71c0 con 0x7f16800ca770 2024-05-27T06:47:25.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167f400640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16800cb7c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1670009870 tx=0x7f167001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebbe2ee87725474 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:25.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167e000640 1 -- 172.21.15.81:0/3245731949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16700a7070 con 0x7f16800ca770 2024-05-27T06:47:25.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167e000640 1 -- 172.21.15.81:0/3245731949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16700b0070 con 0x7f16800ca770 2024-05-27T06:47:25.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.042+0000 7f167e000640 1 -- 172.21.15.81:0/3245731949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16700ab890 con 0x7f16800ca770 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- 172.21.15.81:0/3245731949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 msgr2=0x7f16800cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16800cb7c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1670009870 tx=0x7f167001c560 comp rx=0 tx=0).stop 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- 172.21.15.81:0/3245731949 shutdown_connections 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1680154200 0x7f1680154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 --2- 172.21.15.81:0/3245731949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16800cb7c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- 172.21.15.81:0/3245731949 >> 172.21.15.81:0/3245731949 conn(0x7f1680068ae0 msgr2=0x7f1680068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- 172.21.15.81:0/3245731949 shutdown_connections 2024-05-27T06:47:25.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- 172.21.15.81:0/3245731949 wait complete. 2024-05-27T06:47:25.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 Processor -- start 2024-05-27T06:47:25.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.046+0000 7f1686600640 1 -- start start 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16800cbd00 0x7f1680159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1680154200 0x7f16800c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f16800d0640 con 0x7f16800ca770 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f16800d04c0 con 0x7f1680154200 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1680167590 con 0x7f16800cbd00 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53314/0 (socket says 172.21.15.81:53314) 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 -- 172.21.15.81:0/3851771557 learned_addr learned my addr 172.21.15.81:0/3851771557 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167ea00640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16800cbd00 0x7f1680159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167fe00640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1680154200 0x7f16800c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 -- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16800cbd00 msgr2=0x7f1680159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16800cbd00 0x7f1680159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 -- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1680154200 msgr2=0x7f16800c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1680154200 0x7f16800c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 -- 172.21.15.81:0/3851771557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16801f9780 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f167f400640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f16700ad040 tx=0x7f167001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16700be5f0 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16700be7b0 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16800ce520 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16700beba0 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16800ceae0 con 0x7f16800ca770 2024-05-27T06:47:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f168006a560 con 0x7f16800ca770 2024-05-27T06:47:25.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.050+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f16700b43f0 con 0x7f16800ca770 2024-05-27T06:47:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.054+0000 7f1677400640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 0x7f1660071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.054+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1670129aa0 con 0x7f16800ca770 2024-05-27T06:47:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.054+0000 7f167ea00640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 0x7f1660071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.054+0000 7f167ea00640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 0x7f1660071940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f168014ec60 tx=0x7f166c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:25.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.058+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f16700c4060 con 0x7f16800ca770 2024-05-27T06:47:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1680159eb0 con 0x7f16800ca770 2024-05-27T06:47:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1677400640 1 -- 172.21.15.81:0/3851771557 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f1670016150 con 0x7f16800ca770 2024-05-27T06:47:25.245 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:25.245 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:25.245 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 msgr2=0x7f1660071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 0x7f1660071940 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f168014ec60 tx=0x7f166c009040 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 msgr2=0x7f16801f9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f16700ad040 tx=0x7f167001ada0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 shutdown_connections 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16800cbd00 0x7f1680159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f166006f430 0x7f1660071940 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1680154200 0x7f16800c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 --2- 172.21.15.81:0/3851771557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16800ca770 0x7f16801f9240 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 >> 172.21.15.81:0/3851771557 conn(0x7f1680068ae0 msgr2=0x7f1680068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 shutdown_connections 2024-05-27T06:47:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:25.250+0000 7f1686600640 1 -- 172.21.15.81:0/3851771557 wait complete. 2024-05-27T06:47:25.267 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:27.269 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 Processor -- start 2024-05-27T06:47:27.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 -- start start 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1c700c6fd0 0x7f1c700c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 0x7f1c7015a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c7015db00 con 0x7f1c701546f0 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1c7015e6f0 con 0x7f1c700c6fd0 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c78200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1c700755b0 con 0x7f1c7015a510 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c74e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1c700c6fd0 0x7f1c700c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c75800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1c701546f0 0x7f1c70072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:36000/0 (socket says 172.21.15.81:36000) 2024-05-27T06:47:27.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c75800640 1 -- 172.21.15.81:0/4204746444 learned_addr learned my addr 172.21.15.81:0/4204746444 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c74e00640 1 -- 172.21.15.81:0/4204746444 >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 msgr2=0x7f1c7015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c76200640 1 --2- 172.21.15.81:0/4204746444 >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 0x7f1c7015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c6fe00640 1 -- 172.21.15.81:0/4204746444 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978598757 0 0) 0x7f1c7015db00 con 0x7f1c701546f0 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c74e00640 1 --2- 172.21.15.81:0/4204746444 >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 0x7f1c7015a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c74e00640 1 -- 172.21.15.81:0/4204746444 >> v1:172.21.15.81:6789/0 conn(0x7f1c701546f0 legacy=0x7f1c70072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c74e00640 1 -- 172.21.15.81:0/4204746444 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7015ae60 con 0x7f1c700c6fd0 2024-05-27T06:47:27.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.434+0000 7f1c76200640 1 --2- 172.21.15.81:0/4204746444 >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 0x7f1c7015a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:27.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c74e00640 1 --2- 172.21.15.81:0/4204746444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c700c6fd0 0x7f1c700c73e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c700ca360 tx=0x7f1c5c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814d1e8dd6ef8d93 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:27.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c6fe00640 1 -- 172.21.15.81:0/4204746444 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0b5070 con 0x7f1c700c6fd0 2024-05-27T06:47:27.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c6fe00640 1 -- 172.21.15.81:0/4204746444 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0a8970 con 0x7f1c700c6fd0 2024-05-27T06:47:27.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c6fe00640 1 -- 172.21.15.81:0/4204746444 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0a8d40 con 0x7f1c700c6fd0 2024-05-27T06:47:27.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 -- 172.21.15.81:0/4204746444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c700c6fd0 msgr2=0x7f1c700c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 --2- 172.21.15.81:0/4204746444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c700c6fd0 0x7f1c700c73e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1c700ca360 tx=0x7f1c5c01ec70 comp rx=0 tx=0).stop 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 -- 172.21.15.81:0/4204746444 shutdown_connections 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 --2- 172.21.15.81:0/4204746444 >> v2:172.21.15.106:3300/0 conn(0x7f1c7015a510 0x7f1c7015a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 --2- 172.21.15.81:0/4204746444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c700c6fd0 0x7f1c700c73e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 -- 172.21.15.81:0/4204746444 >> 172.21.15.81:0/4204746444 conn(0x7f1c70070700 msgr2=0x7f1c700cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 -- 172.21.15.81:0/4204746444 shutdown_connections 2024-05-27T06:47:27.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.438+0000 7f1c78200640 1 -- 172.21.15.81:0/4204746444 wait complete. 2024-05-27T06:47:27.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 Processor -- start 2024-05-27T06:47:27.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- start start 2024-05-27T06:47:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 0x7f1c7015d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c701546f0 0x7f1c701f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7015e0a0 con 0x7f1c7015a510 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7015ddc0 con 0x7f1c700c6fd0 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1c7015e380 con 0x7f1c701546f0 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44914/0 (socket says 172.21.15.81:44914) 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 -- 172.21.15.81:0/2704007538 learned_addr learned my addr 172.21.15.81:0/2704007538 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c74e00640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c701546f0 0x7f1c701f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c75800640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 0x7f1c7015d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 -- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c701546f0 msgr2=0x7f1c701f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c701546f0 0x7f1c701f33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 -- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 msgr2=0x7f1c7015d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 0x7f1c7015d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 -- 172.21.15.81:0/2704007538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c701f38e0 con 0x7f1c7015a510 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c75800640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 0x7f1c7015d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c76200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800c5c0 tx=0x7f1c6800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68015070 con 0x7f1c7015a510 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c68011820 con 0x7f1c7015a510 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68011c10 con 0x7f1c7015a510 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7015bd90 con 0x7f1c7015a510 2024-05-27T06:47:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.442+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7015cac0 con 0x7f1c7015a510 2024-05-27T06:47:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.446+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c7015bae0 con 0x7f1c7015a510 2024-05-27T06:47:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.446+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1c68018070 con 0x7f1c7015a510 2024-05-27T06:47:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.446+0000 7f1c6d600640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 0x7f1c50071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.446+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1c6808f1b0 con 0x7f1c7015a510 2024-05-27T06:47:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.446+0000 7f1c75800640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 0x7f1c50071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.450+0000 7f1c75800640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 0x7f1c50071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64007fd0 tx=0x7f1c64005b80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:27.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.450+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1c701f38e0 con 0x7f1c7015a510 2024-05-27T06:47:27.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.642+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c7006a560 con 0x7f1c7015a510 2024-05-27T06:47:27.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.642+0000 7f1c6d600640 1 -- 172.21.15.81:0/2704007538 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6901 (secure 0 0 0) 0x7f1c6805b930 con 0x7f1c7015a510 2024-05-27T06:47:27.639 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:27.639 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:45:34.378415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":4,"incarnation":16,"state":"up:active","state_seq":11,"addr":"172.21.15.106:6833/353223641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":353223641},{"type":"v1","addr":"172.21.15.106:6833","nonce":353223641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4279,4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:27.640 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-05-27T06:47:27.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 msgr2=0x7f1c50071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 0x7f1c50071910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1c64007fd0 tx=0x7f1c64005b80 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 msgr2=0x7f1c7014f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800c5c0 tx=0x7f1c6800ca80 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 shutdown_connections 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1c701546f0 0x7f1c701f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1c5006f400 0x7f1c50071910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c7015a510 0x7f1c7014f650 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 --2- 172.21.15.81:0/2704007538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1c700c6fd0 0x7f1c7015d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 >> 172.21.15.81:0/2704007538 conn(0x7f1c70070700 msgr2=0x7f1c700c9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 shutdown_connections 2024-05-27T06:47:27.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:27.646+0000 7f1c78200640 1 -- 172.21.15.81:0/2704007538 wait complete. 2024-05-27T06:47:27.662 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-05-27T06:47:29.663 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 Processor -- start 2024-05-27T06:47:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 -- start start 2024-05-27T06:47:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 0x7fdc3c072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3c15db00 con 0x7fdc3c0c7510 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fdc3c15e6f0 con 0x7fdc3c0732f0 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.826+0000 7fdc41600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3c0755b0 con 0x7fdc3c1546f0 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44916/0 (socket says 172.21.15.81:44916) 2024-05-27T06:47:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 -- 172.21.15.81:0/1584331347 learned_addr learned my addr 172.21.15.81:0/1584331347 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 -- 172.21.15.81:0/1584331347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 msgr2=0x7fdc3c072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 0x7fdc3c072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 0x7fdc3c072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 -- 172.21.15.81:0/1584331347 >> v1:172.21.15.81:6789/0 conn(0x7fdc3c0c7510 legacy=0x7fdc3c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 -- 172.21.15.81:0/1584331347 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c073860 con 0x7fdc3c0732f0 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3a000640 1 -- 172.21.15.81:0/1584331347 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183068978 0 0) 0x7fdc3c15db00 con 0x7fdc3c0c7510 2024-05-27T06:47:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 0x7fdc3c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc32000640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009870 tx=0x7fdc2401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdb494cebe90d5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3a000640 1 -- 172.21.15.81:0/1584331347 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc240b5070 con 0x7fdc3c0732f0 2024-05-27T06:47:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3a000640 1 -- 172.21.15.81:0/1584331347 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc240a8a50 con 0x7fdc3c0732f0 2024-05-27T06:47:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.830+0000 7fdc3a000640 1 -- 172.21.15.81:0/1584331347 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc240a8e20 con 0x7fdc3c0732f0 2024-05-27T06:47:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- 172.21.15.81:0/1584331347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0732f0 msgr2=0x7fdc3c0c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009870 tx=0x7fdc2401ed50 comp rx=0 tx=0).stop 2024-05-27T06:47:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- 172.21.15.81:0/1584331347 shutdown_connections 2024-05-27T06:47:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c0c6fd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 --2- 172.21.15.81:0/1584331347 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c1546f0 0x7fdc3c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- 172.21.15.81:0/1584331347 >> 172.21.15.81:0/1584331347 conn(0x7fdc3c070700 msgr2=0x7fdc3c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- 172.21.15.81:0/1584331347 shutdown_connections 2024-05-27T06:47:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- 172.21.15.81:0/1584331347 wait complete. 2024-05-27T06:47:29.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 Processor -- start 2024-05-27T06:47:29.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.834+0000 7fdc41600640 1 -- start start 2024-05-27T06:47:29.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c0c7510 0x7fdc3c158720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c15e380 con 0x7fdc3c1546f0 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c15e0a0 con 0x7fdc3c0732f0 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3c154b00 con 0x7fdc3c0c7510 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44918/0 (socket says 172.21.15.81:44918) 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 -- 172.21.15.81:0/4067177584 learned_addr learned my addr 172.21.15.81:0/4067177584 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc32000640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c0c7510 0x7fdc3c158720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 -- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c0c7510 msgr2=0x7fdc3c158720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:29.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c0c7510 0x7fdc3c158720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 -- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 msgr2=0x7fdc3c155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 -- 172.21.15.81:0/4067177584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c158c60 con 0x7fdc3c1546f0 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc3b400640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c790 tx=0x7fdc3400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34003db0 con 0x7fdc3c1546f0 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3400ce40 con 0x7fdc3c1546f0 2024-05-27T06:47:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c1502e0 con 0x7fdc3c1546f0 2024-05-27T06:47:29.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc340052b0 con 0x7fdc3c1546f0 2024-05-27T06:47:29.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1508a0 con 0x7fdc3c1546f0 2024-05-27T06:47:29.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.838+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c06a560 con 0x7fdc3c1546f0 2024-05-27T06:47:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.842+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdc34005450 con 0x7fdc3c1546f0 2024-05-27T06:47:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.842+0000 7fdc33400640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 0x7fdc18071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.842+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdc3408f160 con 0x7fdc3c1546f0 2024-05-27T06:47:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.842+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 0x7fdc18071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.842+0000 7fdc3aa00640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 0x7fdc18071910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00a9b0 tx=0x7fdc2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:29.846+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdc3405c000 con 0x7fdc3c1546f0 2024-05-27T06:47:30.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.038+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc3c0c8e20 con 0x7fdc3c1546f0 2024-05-27T06:47:30.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.038+0000 7fdc33400640 1 -- 172.21.15.81:0/4067177584 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6073 (secure 0 0 0) 0x7fdc3405b900 con 0x7fdc3c1546f0 2024-05-27T06:47:30.035 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:30.036 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:28.290855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:30.036 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 18 2024-05-27T06:47:30.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 msgr2=0x7fdc18071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 0x7fdc18071910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00a9b0 tx=0x7fdc2c008040 comp rx=0 tx=0).stop 2024-05-27T06:47:30.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 msgr2=0x7fdc3c156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c790 tx=0x7fdc3400cc50 comp rx=0 tx=0).stop 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 shutdown_connections 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdc3c0c7510 0x7fdc3c158720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc1806f400 0x7fdc18071910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc3c1546f0 0x7fdc3c156650 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 --2- 172.21.15.81:0/4067177584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc3c0732f0 0x7fdc3c155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 >> 172.21.15.81:0/4067177584 conn(0x7fdc3c070700 msgr2=0x7fdc3c1515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 shutdown_connections 2024-05-27T06:47:30.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.042+0000 7fdc41600640 1 -- 172.21.15.81:0/4067177584 wait complete. 2024-05-27T06:47:30.057 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-05-27T06:47:30.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:47:30.058 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:30.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 Processor -- start 2024-05-27T06:47:30.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 -- start start 2024-05-27T06:47:30.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc640072430 0x7fc640159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc6400c60a0 0x7fc64015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fc6400c5040 con 0x7fc640072430 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc6400c75c0 con 0x7fc6400c60a0 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc644a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc6400c7dd0 con 0x7fc64015a040 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc640072430 0x7fc640159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc640072430 0x7fc640159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44926/0 (socket says 172.21.15.81:44926) 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 -- 172.21.15.81:0/2252353689 learned_addr learned my addr 172.21.15.81:0/2252353689 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63d400640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63e800640 1 --2- 172.21.15.81:0/2252353689 >> v2:172.21.15.106:3300/0 conn(0x7fc6400c60a0 0x7fc64015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:30.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 -- 172.21.15.81:0/2252353689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 msgr2=0x7fc6400c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:30.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 -- 172.21.15.81:0/2252353689 >> v2:172.21.15.106:3300/0 conn(0x7fc6400c60a0 msgr2=0x7fc64015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:30.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 --2- 172.21.15.81:0/2252353689 >> v2:172.21.15.106:3300/0 conn(0x7fc6400c60a0 0x7fc64015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.210+0000 7fc63de00640 1 -- 172.21.15.81:0/2252353689 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc640072840 con 0x7fc640072430 2024-05-27T06:47:30.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc63de00640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc640159b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc64006f9a0 tx=0x7fc63001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa708e2add82b64 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:30.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc63ca00640 1 -- 172.21.15.81:0/2252353689 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300b0070 con 0x7fc640072430 2024-05-27T06:47:30.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc63ca00640 1 -- 172.21.15.81:0/2252353689 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6300ab600 con 0x7fc640072430 2024-05-27T06:47:30.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc63ca00640 1 -- 172.21.15.81:0/2252353689 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300ab9d0 con 0x7fc640072430 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 -- 172.21.15.81:0/2252353689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 msgr2=0x7fc640159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc640159b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc64006f9a0 tx=0x7fc63001c260 comp rx=0 tx=0).stop 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 -- 172.21.15.81:0/2252353689 shutdown_connections 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 --2- 172.21.15.81:0/2252353689 >> v2:172.21.15.106:3300/0 conn(0x7fc6400c60a0 0x7fc64015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 --2- 172.21.15.81:0/2252353689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc640159b00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 -- 172.21.15.81:0/2252353689 >> 172.21.15.81:0/2252353689 conn(0x7fc640070700 msgr2=0x7fc6400cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.214+0000 7fc644a00640 1 -- 172.21.15.81:0/2252353689 shutdown_connections 2024-05-27T06:47:30.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 -- 172.21.15.81:0/2252353689 wait complete. 2024-05-27T06:47:30.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 Processor -- start 2024-05-27T06:47:30.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 -- start start 2024-05-27T06:47:30.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6400c60a0 0x7fc640152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6400ce6f0 con 0x7fc640072430 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401595e0 con 0x7fc6400c60a0 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc644a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc6400c5610 con 0x7fc64015a040 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc63de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc63de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44934/0 (socket says 172.21.15.81:44934) 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.218+0000 7fc63de00640 1 -- 172.21.15.81:0/3306989183 learned_addr learned my addr 172.21.15.81:0/3306989183 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 -- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 msgr2=0x7fc6400ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 -- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6400c60a0 msgr2=0x7fc640152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6400c60a0 0x7fc640152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 -- 172.21.15.81:0/3306989183 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6401f6a90 con 0x7fc640072430 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc63de00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6300ae040 tx=0x7fc630019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300b0040 con 0x7fc640072430 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6300b9070 con 0x7fc640072430 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6300b48a0 con 0x7fc640072430 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc640149530 con 0x7fc640072430 2024-05-27T06:47:30.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6401f9d30 con 0x7fc640072430 2024-05-27T06:47:30.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc64006a560 con 0x7fc640072430 2024-05-27T06:47:30.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc6300ab4c0 con 0x7fc640072430 2024-05-27T06:47:30.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.222+0000 7fc636000640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 0x7fc620071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:30.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.226+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc630129150 con 0x7fc640072430 2024-05-27T06:47:30.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.226+0000 7fc63d400640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 0x7fc620071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:30.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.226+0000 7fc63d400640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 0x7fc620071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc62c006aa0 tx=0x7fc62c006a30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:30.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.230+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc630016110 con 0x7fc640072430 2024-05-27T06:47:30.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.418+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6401f6200 con 0x7fc640072430 2024-05-27T06:47:30.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.418+0000 7fc636000640 1 -- 172.21.15.81:0/3306989183 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6073 (secure 0 0 0) 0x7fc6300f5ed0 con 0x7fc640072430 2024-05-27T06:47:30.415 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:30.415 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:28.290855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:30.415 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 18 2024-05-27T06:47:30.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 msgr2=0x7fc620071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 0x7fc620071940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc62c006aa0 tx=0x7fc62c006a30 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 msgr2=0x7fc6401f6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6300ae040 tx=0x7fc630019000 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 shutdown_connections 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc64015a040 0x7fc6400ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc62006f430 0x7fc620071940 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6400c60a0 0x7fc640152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 --2- 172.21.15.81:0/3306989183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc640072430 0x7fc6401f6550 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 >> 172.21.15.81:0/3306989183 conn(0x7fc640070700 msgr2=0x7fc6400c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:30.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 shutdown_connections 2024-05-27T06:47:30.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:30.422+0000 7fc644a00640 1 -- 172.21.15.81:0/3306989183 wait complete. 2024-05-27T06:47:30.438 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4235": { "addr": "172.21.15.81:6838/1291517441", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 1291517441, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 1291517441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 18, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4281": { "addr": "172.21.15.81:6835/2990302599", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2990302599, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2990302599, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:47:28.290855+0000", "qdb_cluster": [ 4281, 4285, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 4281, "mds_4": 4235 } } } ], "standbys": [] } 2024-05-27T06:47:31.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:47:31.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:47:32.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:32.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 Processor -- start 2024-05-27T06:47:32.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 -- start start 2024-05-27T06:47:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 0x7f70941670e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70940c99f0 con 0x7f70940cc830 2024-05-27T06:47:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7094156fd0 con 0x7f7094156bc0 2024-05-27T06:47:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f709a000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f709415c980 con 0x7f7094160580 2024-05-27T06:47:32.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f7093400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f7093e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 0x7f70941670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f7093400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:52124/0 (socket says 172.21.15.81:52124) 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.594+0000 7f7093400640 1 -- 172.21.15.81:0/2112477596 learned_addr learned my addr 172.21.15.81:0/2112477596 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093e00640 1 -- 172.21.15.81:0/2112477596 >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 msgr2=0x7f70940cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093e00640 1 --2- 172.21.15.81:0/2112477596 >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093e00640 1 -- 172.21.15.81:0/2112477596 >> v1:172.21.15.106:6789/0 conn(0x7f70940cc830 legacy=0x7f709415fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093e00640 1 -- 172.21.15.81:0/2112477596 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709415cc60 con 0x7f7094160580 2024-05-27T06:47:32.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093400640 1 --2- 172.21.15.81:0/2112477596 >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:47:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7093e00640 1 --2- 172.21.15.81:0/2112477596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 0x7f70941670e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f708c009d90 tx=0x7f708c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f03cde169744926 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7092a00640 1 -- 172.21.15.81:0/2112477596 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b4070 con 0x7f7094160580 2024-05-27T06:47:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7092a00640 1 -- 172.21.15.81:0/2112477596 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0af600 con 0x7f7094160580 2024-05-27T06:47:32.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.598+0000 7f7092a00640 1 -- 172.21.15.81:0/2112477596 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0af9d0 con 0x7f7094160580 2024-05-27T06:47:32.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- 172.21.15.81:0/2112477596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 msgr2=0x7f70941670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 --2- 172.21.15.81:0/2112477596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 0x7f70941670e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f708c009d90 tx=0x7f708c01f150 comp rx=0 tx=0).stop 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- 172.21.15.81:0/2112477596 shutdown_connections 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 --2- 172.21.15.81:0/2112477596 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094160580 0x7f70941670e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 --2- 172.21.15.81:0/2112477596 >> v2:172.21.15.106:3300/0 conn(0x7f7094156bc0 0x7f70940cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- 172.21.15.81:0/2112477596 >> 172.21.15.81:0/2112477596 conn(0x7f7094068cb0 msgr2=0x7f70940745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- 172.21.15.81:0/2112477596 shutdown_connections 2024-05-27T06:47:32.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- 172.21.15.81:0/2112477596 wait complete. 2024-05-27T06:47:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 Processor -- start 2024-05-27T06:47:32.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.602+0000 7f709a000640 1 -- start start 2024-05-27T06:47:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094156bc0 0x7f7094153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 0x7f7094150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f709415d500 con 0x7f70940cc830 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f709415d220 con 0x7f7094160580 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7094075350 con 0x7f7094156bc0 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44936/0 (socket says 172.21.15.81:44936) 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 -- 172.21.15.81:0/4293498446 learned_addr learned my addr 172.21.15.81:0/4293498446 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093e00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 0x7f7094150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f708aa00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094156bc0 0x7f7094153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 -- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094156bc0 msgr2=0x7f7094153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094156bc0 0x7f7094153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 -- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 msgr2=0x7f7094150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 0x7f7094150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 -- 172.21.15.81:0/4293498446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70941541a0 con 0x7f70940cc830 2024-05-27T06:47:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093e00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 0x7f7094150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f7093400640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f708400e960 tx=0x7f708400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7084017070 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708400ca00 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70940cee90 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708400cdb0 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70940cf450 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f709406a560 con 0x7f70940cc830 2024-05-27T06:47:32.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.606+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f708401a070 con 0x7f70940cc830 2024-05-27T06:47:32.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.610+0000 7f708be00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 0x7f7070071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:32.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.610+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f708408f2b0 con 0x7f70940cc830 2024-05-27T06:47:32.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.610+0000 7f708aa00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 0x7f7070071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:32.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.610+0000 7f708aa00640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 0x7f7070071910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70940c6640 tx=0x7f707c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:32.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.614+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f708405c030 con 0x7f70940cc830 2024-05-27T06:47:32.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.806+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70940caaf0 con 0x7f70940cc830 2024-05-27T06:47:32.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.806+0000 7f708be00640 1 -- 172.21.15.81:0/4293498446 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6073 (secure 0 0 0) 0x7f708405b930 con 0x7f70940cc830 2024-05-27T06:47:32.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:32.803 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:28.290855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:32.803 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 18 2024-05-27T06:47:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 msgr2=0x7f7070071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 0x7f7070071910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70940c6640 tx=0x7f707c009040 comp rx=0 tx=0).stop 2024-05-27T06:47:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 msgr2=0x7f70940cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f708400e960 tx=0x7f708400ee20 comp rx=0 tx=0).stop 2024-05-27T06:47:32.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 shutdown_connections 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7094156bc0 0x7f7094153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f707006f400 0x7f7070071910 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7094160580 0x7f7094150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 --2- 172.21.15.81:0/4293498446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70940cc830 0x7f70940cd6c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 >> 172.21.15.81:0/4293498446 conn(0x7f7094068cb0 msgr2=0x7f709415f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 shutdown_connections 2024-05-27T06:47:32.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:32.810+0000 7f709a000640 1 -- 172.21.15.81:0/4293498446 wait complete. 2024-05-27T06:47:32.824 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:34.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:34.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 Processor -- start 2024-05-27T06:47:34.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 -- start start 2024-05-27T06:47:34.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 0x7f1e90159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:34.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 0x7f1e900c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:34.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e90072840 con 0x7f1e900c60a0 2024-05-27T06:47:34.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1e900c5040 con 0x7f1e9015a040 2024-05-27T06:47:34.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e98000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e900c75c0 con 0x7f1e90072430 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e94c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 0x7f1e900c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e95600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 0x7f1e90159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e94c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 0x7f1e900c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:52140/0 (socket says 172.21.15.81:52140) 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e94c00640 1 -- 172.21.15.81:0/2106510643 learned_addr learned my addr 172.21.15.81:0/2106510643 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e95600640 1 -- 172.21.15.81:0/2106510643 >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 msgr2=0x7f1e900c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:34.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e95600640 1 --2- 172.21.15.81:0/2106510643 >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 0x7f1e900c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.990+0000 7f1e95600640 1 -- 172.21.15.81:0/2106510643 >> v1:172.21.15.106:6789/0 conn(0x7f1e900c60a0 legacy=0x7f1e9015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:34.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e95600640 1 -- 172.21.15.81:0/2106510643 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e900c5270 con 0x7f1e90072430 2024-05-27T06:47:34.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e95600640 1 --2- 172.21.15.81:0/2106510643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 0x7f1e90159b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1e840097c0 tx=0x7f1e8401e6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81c9aefc38d9d984 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:34.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e8fe00640 1 -- 172.21.15.81:0/2106510643 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e840b3070 con 0x7f1e90072430 2024-05-27T06:47:34.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e8fe00640 1 -- 172.21.15.81:0/2106510643 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e840aa050 con 0x7f1e90072430 2024-05-27T06:47:34.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e8fe00640 1 -- 172.21.15.81:0/2106510643 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e840ae860 con 0x7f1e90072430 2024-05-27T06:47:34.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e98000640 1 -- 172.21.15.81:0/2106510643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 msgr2=0x7f1e90159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.994+0000 7f1e98000640 1 --2- 172.21.15.81:0/2106510643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 0x7f1e90159b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1e840097c0 tx=0x7f1e8401e6b0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- 172.21.15.81:0/2106510643 shutdown_connections 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 --2- 172.21.15.81:0/2106510643 >> v2:172.21.15.106:3300/0 conn(0x7f1e9015a040 0x7f1e900c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 --2- 172.21.15.81:0/2106510643 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e90072430 0x7f1e90159b00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- 172.21.15.81:0/2106510643 >> 172.21.15.81:0/2106510643 conn(0x7f1e90070700 msgr2=0x7f1e900cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:34.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- 172.21.15.81:0/2106510643 shutdown_connections 2024-05-27T06:47:34.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- 172.21.15.81:0/2106510643 wait complete. 2024-05-27T06:47:34.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 Processor -- start 2024-05-27T06:47:34.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- start start 2024-05-27T06:47:34.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:34.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e900c60a0 0x7f1e901f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:34.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e9015a040 0x7f1e90152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:34.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e900ce6f0 con 0x7f1e90072430 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e900c7dd0 con 0x7f1e9015a040 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:34.998+0000 7f1e98000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e900c5610 con 0x7f1e900c60a0 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44944/0 (socket says 172.21.15.81:44944) 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 -- 172.21.15.81:0/4273498107 learned_addr learned my addr 172.21.15.81:0/4273498107 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e94c00640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e900c60a0 0x7f1e901f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e96000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e9015a040 0x7f1e90152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 -- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e900c60a0 msgr2=0x7f1e901f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e900c60a0 0x7f1e901f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 -- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e9015a040 msgr2=0x7f1e90152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e9015a040 0x7f1e90152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 -- 172.21.15.81:0/4273498107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e901f69c0 con 0x7f1e90072430 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e95600640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e8401cd60 tx=0x7f1e8401cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e840be070 con 0x7f1e90072430 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e840b5050 con 0x7f1e90072430 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e840b3030 con 0x7f1e90072430 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e901f5f80 con 0x7f1e90072430 2024-05-27T06:47:34.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e901f9c50 con 0x7f1e90072430 2024-05-27T06:47:35.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e9006a560 con 0x7f1e90072430 2024-05-27T06:47:35.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.002+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e840ad3e0 con 0x7f1e90072430 2024-05-27T06:47:35.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.006+0000 7f1e8d600640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 0x7f1e70071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:35.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.006+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1e84128910 con 0x7f1e90072430 2024-05-27T06:47:35.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.006+0000 7f1e94c00640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 0x7f1e70071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:35.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.006+0000 7f1e94c00640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 0x7f1e70071910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e9013d080 tx=0x7f1e7c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:35.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.010+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1e840182a0 con 0x7f1e90072430 2024-05-27T06:47:35.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.198+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e900c5940 con 0x7f1e90072430 2024-05-27T06:47:35.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.198+0000 7f1e8d600640 1 -- 172.21.15.81:0/4273498107 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6075 (secure 0 0 0) 0x7f1e840aee20 con 0x7f1e90072430 2024-05-27T06:47:35.195 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:35.195 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:33.303745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:resolve","state_seq":41,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:47:35.195 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 19 2024-05-27T06:47:35.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 msgr2=0x7f1e70071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:35.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 0x7f1e70071910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e9013d080 tx=0x7f1e7c009240 comp rx=0 tx=0).stop 2024-05-27T06:47:35.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 msgr2=0x7f1e90151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1e8401cd60 tx=0x7f1e8401cef0 comp rx=0 tx=0).stop 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 shutdown_connections 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e900c60a0 0x7f1e901f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7006f400 0x7f1e70071910 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e9015a040 0x7f1e90152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 --2- 172.21.15.81:0/4273498107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e90072430 0x7f1e90151e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:35.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 >> 172.21.15.81:0/4273498107 conn(0x7f1e90070700 msgr2=0x7f1e90073ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:35.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 shutdown_connections 2024-05-27T06:47:35.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:35.202+0000 7f1e98000640 1 -- 172.21.15.81:0/4273498107 wait complete. 2024-05-27T06:47:35.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:37.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:37.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 Processor -- start 2024-05-27T06:47:37.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 -- start start 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa76c0cb010 0x7fa76c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa76c0c89a0 0x7fa76c072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa76c160130 con 0x7fa76c15fd20 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa76c0c9070 con 0x7fa76c0cb010 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.382+0000 7fa772200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa76c074890 con 0x7fa76c0c89a0 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa76c0cb010 0x7fa76c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa76c0cb010 0x7fa76c156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49450/0 (socket says 172.21.15.81:49450) 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 -- 172.21.15.81:0/92275290 learned_addr learned my addr 172.21.15.81:0/92275290 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:37.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa76be00640 1 --2- 172.21.15.81:0/92275290 >> v2:172.21.15.106:3300/0 conn(0x7fa76c0c89a0 0x7fa76c072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 -- 172.21.15.81:0/92275290 >> v2:172.21.15.106:3300/0 conn(0x7fa76c0c89a0 msgr2=0x7fa76c072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 --2- 172.21.15.81:0/92275290 >> v2:172.21.15.106:3300/0 conn(0x7fa76c0c89a0 0x7fa76c072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 -- 172.21.15.81:0/92275290 >> v1:172.21.15.106:6789/0 conn(0x7fa76c15fd20 legacy=0x7fa76c0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 -- 172.21.15.81:0/92275290 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c0c94c0 con 0x7fa76c0cb010 2024-05-27T06:47:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa762a00640 1 --2- 172.21.15.81:0/92275290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c0cb010 0x7fa76c156a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0cb4c0 tx=0x7fa75401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db4eba28ae19ac7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa76aa00640 1 -- 172.21.15.81:0/92275290 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7540b0070 con 0x7fa76c0cb010 2024-05-27T06:47:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa76aa00640 1 -- 172.21.15.81:0/92275290 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7540ab600 con 0x7fa76c0cb010 2024-05-27T06:47:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.386+0000 7fa76aa00640 1 -- 172.21.15.81:0/92275290 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7540ab9d0 con 0x7fa76c0cb010 2024-05-27T06:47:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- 172.21.15.81:0/92275290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c0cb010 msgr2=0x7fa76c156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 --2- 172.21.15.81:0/92275290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c0cb010 0x7fa76c156a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0cb4c0 tx=0x7fa75401c520 comp rx=0 tx=0).stop 2024-05-27T06:47:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- 172.21.15.81:0/92275290 shutdown_connections 2024-05-27T06:47:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 --2- 172.21.15.81:0/92275290 >> v2:172.21.15.106:3300/0 conn(0x7fa76c0c89a0 0x7fa76c072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 --2- 172.21.15.81:0/92275290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c0cb010 0x7fa76c156a30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- 172.21.15.81:0/92275290 >> 172.21.15.81:0/92275290 conn(0x7fa76c06f230 msgr2=0x7fa76c070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- 172.21.15.81:0/92275290 shutdown_connections 2024-05-27T06:47:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- 172.21.15.81:0/92275290 wait complete. 2024-05-27T06:47:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 Processor -- start 2024-05-27T06:47:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.390+0000 7fa772200640 1 -- start start 2024-05-27T06:47:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa76c0c89a0 0x7fa76c14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 0x7fa76c147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c0c9760 con 0x7fa76c15fd20 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c073fe0 con 0x7fa76c0c89a0 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa76c1698d0 con 0x7fa76c155600 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49458/0 (socket says 172.21.15.81:49458) 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 -- 172.21.15.81:0/1545929428 learned_addr learned my addr 172.21.15.81:0/1545929428 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa76b400640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa76c0c89a0 0x7fa76c14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa76be00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 0x7fa76c147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 -- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 msgr2=0x7fa76c147160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 0x7fa76c147160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 -- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa76c0c89a0 msgr2=0x7fa76c14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa76c0c89a0 0x7fa76c14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 -- 172.21.15.81:0/1545929428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c1fa4a0 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa76be00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 0x7fa76c147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa762a00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa754019130 tx=0x7fa7540192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7540b0040 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7540aa8d0 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7540aacc0 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c154840 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c154e00 con 0x7fa76c15fd20 2024-05-27T06:47:37.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.394+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa76c06a560 con 0x7fa76c15fd20 2024-05-27T06:47:37.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.398+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa7540ab4c0 con 0x7fa76c15fd20 2024-05-27T06:47:37.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.398+0000 7fa763e00640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 0x7fa748071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:37.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.398+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa7541290f0 con 0x7fa76c15fd20 2024-05-27T06:47:37.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.398+0000 7fa76b400640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 0x7fa748071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:37.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.398+0000 7fa76b400640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 0x7fa748071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa75c007fd0 tx=0x7fa75c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:37.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.402+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa754016110 con 0x7fa76c15fd20 2024-05-27T06:47:37.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.590+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa76c0cbf40 con 0x7fa76c15fd20 2024-05-27T06:47:37.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa763e00640 1 -- 172.21.15.81:0/1545929428 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6075 (secure 0 0 0) 0x7fa7540f59c0 con 0x7fa76c15fd20 2024-05-27T06:47:37.588 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:37.588 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:33.303745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:resolve","state_seq":41,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:47:37.589 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 19 2024-05-27T06:47:37.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 msgr2=0x7fa748071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 0x7fa748071910 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa75c007fd0 tx=0x7fa75c009040 comp rx=0 tx=0).stop 2024-05-27T06:47:37.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 msgr2=0x7fa76c146c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:37.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa754019130 tx=0x7fa7540192c0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 shutdown_connections 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa76c155600 0x7fa76c147160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa74806f400 0x7fa748071910 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa76c15fd20 0x7fa76c146c20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 --2- 172.21.15.81:0/1545929428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa76c0c89a0 0x7fa76c14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 >> 172.21.15.81:0/1545929428 conn(0x7fa76c06f230 msgr2=0x7fa76c0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 shutdown_connections 2024-05-27T06:47:37.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:37.594+0000 7fa772200640 1 -- 172.21.15.81:0/1545929428 wait complete. 2024-05-27T06:47:37.614 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:39.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:39.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.778+0000 7f192e600640 1 Processor -- start 2024-05-27T06:47:39.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.778+0000 7f192e600640 1 -- start start 2024-05-27T06:47:39.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f192e600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f192e600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f192e600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19280c9f80 con 0x7f19280c8880 2024-05-27T06:47:39.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f192e600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f19280c58b0 con 0x7f192815cb80 2024-05-27T06:47:39.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f192e600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f19280c83f0 con 0x7f192806c790 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1927400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1927e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f19280c8880 0x7f19280c8c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:41378/0 (socket says 172.21.15.81:41378) 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1927400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:44012/0 (socket says 172.21.15.81:44012) 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1927400640 1 -- 172.21.15.81:0/4288774460 learned_addr learned my addr 172.21.15.81:0/4288774460 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:46712/0 (socket says 172.21.15.81:46712) 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 -- 172.21.15.81:0/4288774460 >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 msgr2=0x7f1928152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:39.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 --2- 172.21.15.81:0/4288774460 >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 -- 172.21.15.81:0/4288774460 >> v1:172.21.15.106:6789/0 conn(0x7f19280c8880 legacy=0x7f19280c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1926a00640 1 -- 172.21.15.81:0/4288774460 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19280c52f0 con 0x7f192806c790 2024-05-27T06:47:39.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.782+0000 7f1927400640 1 --2- 172.21.15.81:0/4288774460 >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:47:39.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f1926a00640 1 --2- 172.21.15.81:0/4288774460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19140097c0 tx=0x7f191401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09d3efea4b3c721 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:39.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f1926000640 1 -- 172.21.15.81:0/4288774460 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140b0070 con 0x7f192806c790 2024-05-27T06:47:39.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f1926000640 1 -- 172.21.15.81:0/4288774460 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19140ab600 con 0x7f192806c790 2024-05-27T06:47:39.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f1926000640 1 -- 172.21.15.81:0/4288774460 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140ab9d0 con 0x7f192806c790 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 -- 172.21.15.81:0/4288774460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 msgr2=0x7f192806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 --2- 172.21.15.81:0/4288774460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19140097c0 tx=0x7f191401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 -- 172.21.15.81:0/4288774460 shutdown_connections 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 --2- 172.21.15.81:0/4288774460 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192806c790 0x7f192806cba0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 --2- 172.21.15.81:0/4288774460 >> v2:172.21.15.106:3300/0 conn(0x7f192815cb80 0x7f1928152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 -- 172.21.15.81:0/4288774460 >> 172.21.15.81:0/4288774460 conn(0x7f1928070700 msgr2=0x7f19280cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:39.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 -- 172.21.15.81:0/4288774460 shutdown_connections 2024-05-27T06:47:39.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.786+0000 7f192e600640 1 -- 172.21.15.81:0/4288774460 wait complete. 2024-05-27T06:47:39.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 Processor -- start 2024-05-27T06:47:39.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- start start 2024-05-27T06:47:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19280c8880 0x7f192814bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192815cb80 0x7f1928146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19280c9c80 con 0x7f192806c790 2024-05-27T06:47:39.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f19280c9b00 con 0x7f19280c8880 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1928167810 con 0x7f192815cb80 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49470/0 (socket says 172.21.15.81:49470) 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 -- 172.21.15.81:0/817960177 learned_addr learned my addr 172.21.15.81:0/817960177 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1926a00640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19280c8880 0x7f192814bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927e00640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192815cb80 0x7f1928146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 -- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192815cb80 msgr2=0x7f1928146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192815cb80 0x7f1928146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 -- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19280c8880 msgr2=0x7f192814bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19280c8880 0x7f192814bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 -- 172.21.15.81:0/817960177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1928147cf0 con 0x7f192806c790 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f1927400640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f191800e960 tx=0x7f191800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:39.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1918017070 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191800ca00 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191800cdb0 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1928146ff0 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.790+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1928148b30 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f192806a560 con 0x7f192806c790 2024-05-27T06:47:39.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f191801a070 con 0x7f192806c790 2024-05-27T06:47:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f191f400640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 0x7f1908071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f191808f190 con 0x7f192806c790 2024-05-27T06:47:39.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f1926a00640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 0x7f1908071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:39.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.794+0000 7f1926a00640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 0x7f1908071940 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19140ad000 tx=0x7f1914008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:39.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.798+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f191805b960 con 0x7f192806c790 2024-05-27T06:47:39.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.990+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19280c41e0 con 0x7f192806c790 2024-05-27T06:47:39.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.990+0000 7f191f400640 1 -- 172.21.15.81:0/817960177 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6077 (secure 0 0 0) 0x7f191805fc60 con 0x7f192806c790 2024-05-27T06:47:39.986 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:39.986 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:38.872090+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":43,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:39.987 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-05-27T06:47:39.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 msgr2=0x7f1908071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 0x7f1908071940 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f19140ad000 tx=0x7f1914008040 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 msgr2=0x7f19281477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f191800e960 tx=0x7f191800ee20 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 shutdown_connections 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f192815cb80 0x7f1928146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f190806f430 0x7f1908071940 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19280c8880 0x7f192814bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 --2- 172.21.15.81:0/817960177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192806c790 0x7f19281477b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 >> 172.21.15.81:0/817960177 conn(0x7f1928070700 msgr2=0x7f19280cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 shutdown_connections 2024-05-27T06:47:39.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:39.994+0000 7f192e600640 1 -- 172.21.15.81:0/817960177 wait complete. 2024-05-27T06:47:40.009 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:42.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:42.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 Processor -- start 2024-05-27T06:47:42.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 -- start start 2024-05-27T06:47:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 0x7fdcb80cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb80755a0 con 0x7fdcb815a3e0 2024-05-27T06:47:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb80c7400 con 0x7fdcb80cc930 2024-05-27T06:47:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.174+0000 7fdcbce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb80ca9f0 con 0x7fdcb80c6de0 2024-05-27T06:47:42.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdcb815a3e0 0x7fdcb80ca2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52424/0 (socket says 172.21.15.81:52424) 2024-05-27T06:47:42.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5400640 1 -- 172.21.15.81:0/2877633914 learned_addr learned my addr 172.21.15.81:0/2877633914 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:42.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/2877633914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 0x7fdcb80cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:42.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb4a00640 1 -- 172.21.15.81:0/2877633914 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 756105255 0 0) 0x7fdcb80755a0 con 0x7fdcb815a3e0 2024-05-27T06:47:42.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb4a00640 1 -- 172.21.15.81:0/2877633914 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc9c003530 con 0x7fdcb815a3e0 2024-05-27T06:47:42.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5e00640 1 -- 172.21.15.81:0/2877633914 >> v1:172.21.15.106:6789/0 conn(0x7fdcb80cc930 legacy=0x7fdcb8164d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5e00640 1 -- 172.21.15.81:0/2877633914 >> v1:172.21.15.81:6789/0 conn(0x7fdcb815a3e0 legacy=0x7fdcb80ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5e00640 1 -- 172.21.15.81:0/2877633914 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb80c98c0 con 0x7fdcb80c6de0 2024-05-27T06:47:42.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.178+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/2877633914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 0x7fdcb80cbd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097f0 tx=0x7fdca801ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efee3168be2689f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:42.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcb4a00640 1 -- 172.21.15.81:0/2877633914 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca80b5070 con 0x7fdcb80c6de0 2024-05-27T06:47:42.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcb4a00640 1 -- 172.21.15.81:0/2877633914 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca80a89f0 con 0x7fdcb80c6de0 2024-05-27T06:47:42.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcb4a00640 1 -- 172.21.15.81:0/2877633914 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca80a8dc0 con 0x7fdcb80c6de0 2024-05-27T06:47:42.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- 172.21.15.81:0/2877633914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 msgr2=0x7fdcb80cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 --2- 172.21.15.81:0/2877633914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 0x7fdcb80cbd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097f0 tx=0x7fdca801ecf0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- 172.21.15.81:0/2877633914 shutdown_connections 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 --2- 172.21.15.81:0/2877633914 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb80c6de0 0x7fdcb80cbd40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- 172.21.15.81:0/2877633914 >> 172.21.15.81:0/2877633914 conn(0x7fdcb8068ab0 msgr2=0x7fdcb8068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- 172.21.15.81:0/2877633914 shutdown_connections 2024-05-27T06:47:42.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- 172.21.15.81:0/2877633914 wait complete. 2024-05-27T06:47:42.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 Processor -- start 2024-05-27T06:47:42.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.182+0000 7fdcbce00640 1 -- start start 2024-05-27T06:47:42.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 0x7fdcb8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:42.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:42.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 0x7fdcb814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb815a7f0 con 0x7fdcb80cc930 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb80cb820 con 0x7fdcb80c6de0 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb80c95c0 con 0x7fdcb815a3e0 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49486/0 (socket says 172.21.15.81:49486) 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 -- 172.21.15.81:0/3179367955 learned_addr learned my addr 172.21.15.81:0/3179367955 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 0x7fdcb8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb6800640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 0x7fdcb814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 -- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 msgr2=0x7fdcb814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 0x7fdcb814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 -- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 msgr2=0x7fdcb8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 0x7fdcb8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 -- 172.21.15.81:0/3179367955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb8157370 con 0x7fdcb80cc930 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 0x7fdcb8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb6800640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 0x7fdcb814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcb5400640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4017070 con 0x7fdcb80cc930 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca4005cf0 con 0x7fdcb80cc930 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb8154d70 con 0x7fdcb80cc930 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40072d0 con 0x7fdcb80cc930 2024-05-27T06:47:42.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb81f45f0 con 0x7fdcb80cc930 2024-05-27T06:47:42.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.186+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb806a560 con 0x7fdcb80cc930 2024-05-27T06:47:42.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.190+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdca401a070 con 0x7fdcb80cc930 2024-05-27T06:47:42.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.190+0000 7fdcae000640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 0x7fdc98071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:42.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.190+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdca408e550 con 0x7fdcb80cc930 2024-05-27T06:47:42.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.190+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 0x7fdc98071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:42.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.190+0000 7fdcb5e00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 0x7fdc98071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdca80b2000 tx=0x7fdca8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:42.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.194+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdca405ada0 con 0x7fdcb80cc930 2024-05-27T06:47:42.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.382+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcb80c90c0 con 0x7fdcb80cc930 2024-05-27T06:47:42.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcae000640 1 -- 172.21.15.81:0/3179367955 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6077 (secure 0 0 0) 0x7fdca405f0a0 con 0x7fdcb80cc930 2024-05-27T06:47:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:42.380 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:38.872090+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:reconnect","state_seq":43,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:47:42.381 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-05-27T06:47:42.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 msgr2=0x7fdc98071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 0x7fdc98071940 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdca80b2000 tx=0x7fdca8008040 comp rx=0 tx=0).stop 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 msgr2=0x7fdcb814a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca400c820 comp rx=0 tx=0).stop 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 shutdown_connections 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdcb815a3e0 0x7fdcb814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdc9806f430 0x7fdc98071940 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcb80cc930 0x7fdcb814a690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 --2- 172.21.15.81:0/3179367955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdcb80c6de0 0x7fdcb8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:42.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 >> 172.21.15.81:0/3179367955 conn(0x7fdcb8068ab0 msgr2=0x7fdcb806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:42.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 shutdown_connections 2024-05-27T06:47:42.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:42.386+0000 7fdcbce00640 1 -- 172.21.15.81:0/3179367955 wait complete. 2024-05-27T06:47:42.403 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4235": { "addr": "172.21.15.81:6838/1291517441", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 1291517441, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 1291517441, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 18, "join_fscid": -1, "name": "e", "rank": 4, "state": "up:reconnect", "state_seq": 43 }, "gid_4281": { "addr": "172.21.15.81:6835/2990302599", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2990302599, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2990302599, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:47:38.872090+0000", "qdb_cluster": [ 4281, 4285, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 4281, "mds_4": 4235 } } } ], "standbys": [] } 2024-05-27T06:47:44.404 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:44.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.566+0000 7f5665e00640 1 Processor -- start 2024-05-27T06:47:44.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.566+0000 7f5665e00640 1 -- start start 2024-05-27T06:47:44.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5660160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f566015cb60 con 0x7f56600cebf0 2024-05-27T06:47:44.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f566015ce40 con 0x7f56600c96b0 2024-05-27T06:47:44.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f566015d120 con 0x7f5660160370 2024-05-27T06:47:44.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f56600cebf0 0x7f5660167040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52426/0 (socket says 172.21.15.81:52426) 2024-05-27T06:47:44.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665400640 1 -- 172.21.15.81:0/2707855736 learned_addr learned my addr 172.21.15.81:0/2707855736 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5657e00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5660160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744717066 0 0) 0x7f566015cb60 con 0x7f56600cebf0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5644003530 con 0x7f56600cebf0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1870763438 0 0) 0x7f5644003530 con 0x7f56600cebf0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 msgr2=0x7f5660160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5660160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 >> v1:172.21.15.106:6789/0 conn(0x7f56600c96b0 legacy=0x7f5660156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566015d400 con 0x7f56600cebf0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f5658001a80 con 0x7f56600cebf0 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 >> v1:172.21.15.81:6789/0 conn(0x7f56600cebf0 legacy=0x7f5660167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56440039f0 0x7f5644003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5644004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5644004d50 0x7f5644005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f566015d120 con 0x7f56440039f0 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f564400bb60 con 0x7f5644004d50 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f564400bd10 con 0x7f5660160370 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56440039f0 0x7f5644003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5657e00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5644004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5665400640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5644004d50 0x7f5644005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 -- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 msgr2=0x7f5644004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5660160370 0x7f5644004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 -- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5644004d50 msgr2=0x7f5644005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5644004d50 0x7f5644005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.570+0000 7f5664a00640 1 -- 172.21.15.81:0/2707855736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5644003530 con 0x7f56440039f0 2024-05-27T06:47:44.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5664a00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56440039f0 0x7f5644003e20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5650001840 tx=0x7f565001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376ce226bd34106d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:44.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56500b5070 con 0x7f56440039f0 2024-05-27T06:47:44.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56500a8930 con 0x7f56440039f0 2024-05-27T06:47:44.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f565fe00640 1 -- 172.21.15.81:0/2707855736 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650001460 con 0x7f56440039f0 2024-05-27T06:47:44.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665e00640 1 -- 172.21.15.81:0/2707855736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56440039f0 msgr2=0x7f5644003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665e00640 1 --2- 172.21.15.81:0/2707855736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56440039f0 0x7f5644003e20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5650001840 tx=0x7f565001ec50 comp rx=0 tx=0).stop 2024-05-27T06:47:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665400640 1 -- 172.21.15.81:0/2707855736 reap_dead start 2024-05-27T06:47:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665e00640 1 -- 172.21.15.81:0/2707855736 shutdown_connections 2024-05-27T06:47:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665e00640 1 -- 172.21.15.81:0/2707855736 >> 172.21.15.81:0/2707855736 conn(0x7f5660071390 msgr2=0x7f56600717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.574+0000 7f5665e00640 1 -- 172.21.15.81:0/2707855736 shutdown_connections 2024-05-27T06:47:44.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 -- 172.21.15.81:0/2707855736 wait complete. 2024-05-27T06:47:44.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 Processor -- start 2024-05-27T06:47:44.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 -- start start 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 0x7f566006e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 0x7f56600c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f566015c8a0 con 0x7f566014ceb0 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f566015c720 con 0x7f566015f9b0 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5660074430 con 0x7f56440039f0 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.578+0000 7f5665400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49514/0 (socket says 172.21.15.81:49514) 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 -- 172.21.15.81:0/600166547 learned_addr learned my addr 172.21.15.81:0/600166547 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:44.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5664a00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 0x7f566006e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5657e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 0x7f56600c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 -- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 msgr2=0x7f566006e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 0x7f566006e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 -- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 msgr2=0x7f56600c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 0x7f56600c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 -- 172.21.15.81:0/600166547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566015c470 con 0x7f566014ceb0 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5657e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 0x7f56600c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:44.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5664a00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 0x7f566006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:44.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665400640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5644004790 tx=0x7f5658007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:44.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658017020 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565800ac20 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580133f0 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566006e700 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566006de20 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5660050870 con 0x7f566014ceb0 2024-05-27T06:47:44.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.582+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f565801a020 con 0x7f566014ceb0 2024-05-27T06:47:44.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.586+0000 7f565d600640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 0x7f563c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:44.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.586+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5658090130 con 0x7f566014ceb0 2024-05-27T06:47:44.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.586+0000 7f5664a00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 0x7f563c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:44.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.586+0000 7f5664a00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 0x7f563c071940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f56500af000 tx=0x7f5650001280 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:44.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.586+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5658021020 con 0x7f566014ceb0 2024-05-27T06:47:44.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.778+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5660076320 con 0x7f566014ceb0 2024-05-27T06:47:44.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.778+0000 7f565d600640 1 -- 172.21.15.81:0/600166547 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6074 (secure 0 0 0) 0x7f565805d050 con 0x7f566014ceb0 2024-05-27T06:47:44.774 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:44.774 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:43.926818+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:rejoin","state_seq":45,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:47:44.775 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 21 2024-05-27T06:47:44.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 msgr2=0x7f563c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 0x7f563c071940 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f56500af000 tx=0x7f5650001280 comp rx=0 tx=0).stop 2024-05-27T06:47:44.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 msgr2=0x7f566014c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:44.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5644004790 tx=0x7f5658007bd0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 shutdown_connections 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f563c06f430 0x7f563c071940 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f566014ceb0 0x7f566014c320 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f566015f9b0 0x7f56600c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 --2- 172.21.15.81:0/600166547 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56440039f0 0x7f566006e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 >> 172.21.15.81:0/600166547 conn(0x7f5660071390 msgr2=0x7f566015a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 shutdown_connections 2024-05-27T06:47:44.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:44.782+0000 7f5665e00640 1 -- 172.21.15.81:0/600166547 wait complete. 2024-05-27T06:47:44.796 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:46.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:46.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.962+0000 7fea3a800640 1 Processor -- start 2024-05-27T06:47:46.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.962+0000 7fea3a800640 1 -- start start 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.962+0000 7fea3a800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fea340c5b60 0x7fea340c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.962+0000 7fea3a800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34161cd0 0x7fea341640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea3a800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea34072840 con 0x7fea34072430 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea3a800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fea340c5040 con 0x7fea340c5b60 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea3a800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fea340c75c0 con 0x7fea34161cd0 2024-05-27T06:47:46.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fea340c5b60 0x7fea340c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fea340c5b60 0x7fea340c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53876/0 (socket says 172.21.15.81:53876) 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 -- 172.21.15.81:0/4042889194 learned_addr learned my addr 172.21.15.81:0/4042889194 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea33e00640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34161cd0 0x7fea341640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 -- 172.21.15.81:0/4042889194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34161cd0 msgr2=0x7fea341640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34161cd0 0x7fea341640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 -- 172.21.15.81:0/4042889194 >> v1:172.21.15.106:6789/0 conn(0x7fea34072430 legacy=0x7fea34159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:46.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 -- 172.21.15.81:0/4042889194 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea340c5270 con 0x7fea340c5b60 2024-05-27T06:47:46.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea2aa00640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea340c5f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00bb10 tx=0x7fea1c01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178b0161efe3e621 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:46.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea32a00640 1 -- 172.21.15.81:0/4042889194 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0b4070 con 0x7fea340c5b60 2024-05-27T06:47:46.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea32a00640 1 -- 172.21.15.81:0/4042889194 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c0a7a00 con 0x7fea340c5b60 2024-05-27T06:47:46.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.966+0000 7fea32a00640 1 -- 172.21.15.81:0/4042889194 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0a7dd0 con 0x7fea340c5b60 2024-05-27T06:47:46.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- 172.21.15.81:0/4042889194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 msgr2=0x7fea340c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:46.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea340c5f70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00bb10 tx=0x7fea1c01ec30 comp rx=0 tx=0).stop 2024-05-27T06:47:46.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- 172.21.15.81:0/4042889194 shutdown_connections 2024-05-27T06:47:46.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34161cd0 0x7fea341640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:46.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 --2- 172.21.15.81:0/4042889194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea340c5f70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:46.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- 172.21.15.81:0/4042889194 >> 172.21.15.81:0/4042889194 conn(0x7fea34070700 msgr2=0x7fea340cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:46.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- 172.21.15.81:0/4042889194 shutdown_connections 2024-05-27T06:47:46.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- 172.21.15.81:0/4042889194 wait complete. 2024-05-27T06:47:46.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 Processor -- start 2024-05-27T06:47:46.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.970+0000 7fea3a800640 1 -- start start 2024-05-27T06:47:46.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 0x7fea34155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 0x7fea341584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea340ce6f0 con 0x7fea340c5b60 2024-05-27T06:47:46.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea340c7dd0 con 0x7fea34161cd0 2024-05-27T06:47:46.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fea340c5610 con 0x7fea34072430 2024-05-27T06:47:46.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53880/0 (socket says 172.21.15.81:53880) 2024-05-27T06:47:46.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 -- 172.21.15.81:0/3740811164 learned_addr learned my addr 172.21.15.81:0/3740811164 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea33400640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 0x7fea34155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 -- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 msgr2=0x7fea34155060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea33e00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 0x7fea341584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 0x7fea34155060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:46.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 -- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 msgr2=0x7fea341584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 0x7fea341584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 -- 172.21.15.81:0/3740811164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea341563f0 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea33400640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 0x7fea34155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea33e00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 0x7fea341584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2aa00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea1c01d2e0 tx=0x7fea1c01b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0b4070 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c0b6070 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea34155e30 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0afe60 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea3415d020 con 0x7fea340c5b60 2024-05-27T06:47:46.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.974+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea3406a560 con 0x7fea340c5b60 2024-05-27T06:47:46.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.978+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea1c0c43f0 con 0x7fea340c5b60 2024-05-27T06:47:46.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.978+0000 7fea2be00640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 0x7fea10071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:46.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.978+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fea1c129810 con 0x7fea340c5b60 2024-05-27T06:47:46.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.978+0000 7fea33400640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 0x7fea10071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:46.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.978+0000 7fea33400640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 0x7fea10071910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fea34070fe0 tx=0x7fea24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:46.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:46.982+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fea1c018820 con 0x7fea340c5b60 2024-05-27T06:47:47.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea34152830 con 0x7fea340c5b60 2024-05-27T06:47:47.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea2be00640 1 -- 172.21.15.81:0/3740811164 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6074 (secure 0 0 0) 0x7fea1c0f66c0 con 0x7fea340c5b60 2024-05-27T06:47:47.169 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:47.169 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:43.926818+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:rejoin","state_seq":45,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:47:47.169 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 21 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 msgr2=0x7fea10071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 0x7fea10071910 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fea34070fe0 tx=0x7fea24009040 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 msgr2=0x7fea341f9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fea1c01d2e0 tx=0x7fea1c01b9d0 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 shutdown_connections 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea1006f400 0x7fea10071910 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea34161cd0 0x7fea341584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea340c5b60 0x7fea341f9890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 --2- 172.21.15.81:0/3740811164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea34072430 0x7fea34155060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:47.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 >> 172.21.15.81:0/3740811164 conn(0x7fea34070700 msgr2=0x7fea340c6640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:47.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 shutdown_connections 2024-05-27T06:47:47.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:47.174+0000 7fea3a800640 1 -- 172.21.15.81:0/3740811164 wait complete. 2024-05-27T06:47:47.191 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:47:48.954 INFO:tasks.workunit.client.0.smithi081.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-05-27T06:47:49.192 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:49.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.382+0000 7f7cf5800640 1 Processor -- start 2024-05-27T06:47:49.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.382+0000 7f7cf5800640 1 -- start start 2024-05-27T06:47:49.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf5800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cf00c6a90 0x7f7cf00c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 0x7f7cf00c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf5800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf00c73f0 con 0x7f7cf0075110 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf5800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf0072780 con 0x7f7cf00c6a90 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf5800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf00c5b50 con 0x7f7cf00c6fe0 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cf00c6a90 0x7f7cf00c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cf00c6a90 0x7f7cf00c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53890/0 (socket says 172.21.15.81:53890) 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 -- 172.21.15.81:0/2523855019 learned_addr learned my addr 172.21.15.81:0/2523855019 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf4e00640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 0x7f7cf00c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 -- 172.21.15.81:0/2523855019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 msgr2=0x7f7cf00c9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:49.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 0x7f7cf00c9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 -- 172.21.15.81:0/2523855019 >> v1:172.21.15.106:6789/0 conn(0x7f7cf0075110 legacy=0x7f7cf0153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cefe00640 1 -- 172.21.15.81:0/2523855019 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf0073e40 con 0x7f7cf00c6a90 2024-05-27T06:47:49.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.386+0000 7f7cf4e00640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 0x7f7cf00c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:49.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cefe00640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf00c6a90 0x7f7cf00c64d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009870 tx=0x7f7ce001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c992ba745ae2de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:49.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7ceea00640 1 -- 172.21.15.81:0/2523855019 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b1070 con 0x7f7cf00c6a90 2024-05-27T06:47:49.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7ceea00640 1 -- 172.21.15.81:0/2523855019 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce00a8050 con 0x7f7cf00c6a90 2024-05-27T06:47:49.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7ceea00640 1 -- 172.21.15.81:0/2523855019 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00ac860 con 0x7f7cf00c6a90 2024-05-27T06:47:49.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 -- 172.21.15.81:0/2523855019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf00c6a90 msgr2=0x7f7cf00c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf00c6a90 0x7f7cf00c64d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0009870 tx=0x7f7ce001c4d0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 -- 172.21.15.81:0/2523855019 shutdown_connections 2024-05-27T06:47:49.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6fe0 0x7f7cf00c9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2523855019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf00c6a90 0x7f7cf00c64d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 -- 172.21.15.81:0/2523855019 >> 172.21.15.81:0/2523855019 conn(0x7f7cf006d180 msgr2=0x7f7cf00cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:49.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 -- 172.21.15.81:0/2523855019 shutdown_connections 2024-05-27T06:47:49.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.390+0000 7f7cf5800640 1 -- 172.21.15.81:0/2523855019 wait complete. 2024-05-27T06:47:49.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 Processor -- start 2024-05-27T06:47:49.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 -- start start 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6a90 0x7f7cf0144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 0x7f7cf01496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01646a0 con 0x7f7cf0075110 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf0075520 con 0x7f7cf00c6fe0 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf5800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf0164820 con 0x7f7cf00c6a90 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53904/0 (socket says 172.21.15.81:53904) 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 -- 172.21.15.81:0/2634043736 learned_addr learned my addr 172.21.15.81:0/2634043736 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cef400640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6a90 0x7f7cf0144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf4e00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 0x7f7cf01496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 -- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6a90 msgr2=0x7f7cf0144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6a90 0x7f7cf0144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 -- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 msgr2=0x7f7cf01496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 0x7f7cf01496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cefe00640 1 -- 172.21.15.81:0/2634043736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf0148040 con 0x7f7cf0075110 2024-05-27T06:47:49.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.394+0000 7f7cf4e00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 0x7f7cf01496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7cefe00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7ce001ab80 tx=0x7f7ce001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b1040 con 0x7f7cf0075110 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce00bc070 con 0x7f7cf0075110 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf0166ec0 con 0x7f7cf0075110 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b7690 con 0x7f7cf0075110 2024-05-27T06:47:49.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf01482b0 con 0x7f7cf0075110 2024-05-27T06:47:49.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf0050870 con 0x7f7cf0075110 2024-05-27T06:47:49.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.398+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7ce00b7830 con 0x7f7cf0075110 2024-05-27T06:47:49.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.402+0000 7f7ce7e00640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 0x7f7cd0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:49.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.402+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7ce0129480 con 0x7f7cf0075110 2024-05-27T06:47:49.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.402+0000 7f7cef400640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 0x7f7cd0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:49.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.402+0000 7f7cef400640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 0x7f7cd0071910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00cdb90 tx=0x7f7cdc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:49.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.406+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7ce0016150 con 0x7f7cf0075110 2024-05-27T06:47:49.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.654+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cf00c5770 con 0x7f7cf0075110 2024-05-27T06:47:49.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.654+0000 7f7ce7e00640 1 -- 172.21.15.81:0/2634043736 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6079 (secure 0 0 0) 0x7f7ce00f6320 con 0x7f7cf0075110 2024-05-27T06:47:49.651 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:49.651 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4235,4287,4281,4285,4302]},"id":1}]} 2024-05-27T06:47:49.651 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 22 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 msgr2=0x7f7cd0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 0x7f7cd0071910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7cf00cdb90 tx=0x7f7cdc009040 comp rx=0 tx=0).stop 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 msgr2=0x7f7cf0148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7ce001ab80 tx=0x7f7ce001ad10 comp rx=0 tx=0).stop 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 shutdown_connections 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7cf00c6a90 0x7f7cf0144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cd006f400 0x7f7cd0071910 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cf00c6fe0 0x7f7cf01496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 --2- 172.21.15.81:0/2634043736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf0075110 0x7f7cf0148ab0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:49.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 >> 172.21.15.81:0/2634043736 conn(0x7f7cf006d180 msgr2=0x7f7cf00c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:49.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 shutdown_connections 2024-05-27T06:47:49.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:49.658+0000 7f7cf5800640 1 -- 172.21.15.81:0/2634043736 wait complete. 2024-05-27T06:47:49.673 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=4, replacing gid=4279 2024-05-27T06:47:49.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2024-05-27T06:47:52.560 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir foo 2024-05-27T06:47:52.571 INFO:tasks.workunit.client.0.smithi081.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-05-27T06:47:53.220 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir foo/.snap/barsnap 2024-05-27T06:47:53.675 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-05-27T06:47:53.675 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-27T06:47:53.675 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-27T06:47:53.679 INFO:tasks.ceph.mds.d:Started 2024-05-27T06:47:53.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:53.781 INFO:tasks.ceph.mds.d.smithi106.stdout:starting mds.d at 2024-05-27T06:47:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 Processor -- start 2024-05-27T06:47:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 -- start start 2024-05-27T06:47:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 0x7f53a8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f53a815a510 0x7f53a815a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a815db00 con 0x7f53a80732f0 2024-05-27T06:47:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f53a815e6f0 con 0x7f53a815a510 2024-05-27T06:47:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53ae600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f53a80755b0 con 0x7f53a81546f0 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53a7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f53a815a510 0x7f53a815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f539ea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f53a80732f0 0x7f53a80c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43032/0 (socket says 172.21.15.81:43032) 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53a7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f53a815a510 0x7f53a815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53918/0 (socket says 172.21.15.81:53918) 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f539ea00640 1 -- 172.21.15.81:0/2150879661 learned_addr learned my addr 172.21.15.81:0/2150879661 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.834+0000 7f53a7400640 1 --2- 172.21.15.81:0/2150879661 >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 0x7f53a8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7e00640 1 -- 172.21.15.81:0/2150879661 >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 msgr2=0x7f53a8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7e00640 1 --2- 172.21.15.81:0/2150879661 >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 0x7f53a8072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:53.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7e00640 1 -- 172.21.15.81:0/2150879661 >> v1:172.21.15.81:6789/0 conn(0x7f53a80732f0 legacy=0x7f53a80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:53.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7e00640 1 -- 172.21.15.81:0/2150879661 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a815ae60 con 0x7f53a815a510 2024-05-27T06:47:53.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7400640 1 --2- 172.21.15.81:0/2150879661 >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 0x7f53a8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:53.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a7e00640 1 --2- 172.21.15.81:0/2150879661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a815a920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f53a806d950 tx=0x7f53a001f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4ade87f1874f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:53.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a6a00640 1 -- 172.21.15.81:0/2150879661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00b4070 con 0x7f53a815a510 2024-05-27T06:47:53.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a6a00640 1 -- 172.21.15.81:0/2150879661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a00af600 con 0x7f53a815a510 2024-05-27T06:47:53.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.838+0000 7f53a6a00640 1 -- 172.21.15.81:0/2150879661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00af9d0 con 0x7f53a815a510 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- 172.21.15.81:0/2150879661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 msgr2=0x7f53a815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 --2- 172.21.15.81:0/2150879661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a815a920 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f53a806d950 tx=0x7f53a001f110 comp rx=0 tx=0).stop 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- 172.21.15.81:0/2150879661 shutdown_connections 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 --2- 172.21.15.81:0/2150879661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a815a920 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 --2- 172.21.15.81:0/2150879661 >> v2:172.21.15.106:3300/0 conn(0x7f53a81546f0 0x7f53a8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- 172.21.15.81:0/2150879661 >> 172.21.15.81:0/2150879661 conn(0x7f53a8070700 msgr2=0x7f53a80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- 172.21.15.81:0/2150879661 shutdown_connections 2024-05-27T06:47:53.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- 172.21.15.81:0/2150879661 wait complete. 2024-05-27T06:47:53.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 Processor -- start 2024-05-27T06:47:53.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 -- start start 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a80732f0 0x7f53a81f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.842+0000 7f53ae600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a81546f0 0x7f53a814dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53a815e0a0 con 0x7f53a815a510 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f53a815ddc0 con 0x7f53a81546f0 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f53a815e380 con 0x7f53a80732f0 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53922/0 (socket says 172.21.15.81:53922) 2024-05-27T06:47:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 -- 172.21.15.81:0/4264543636 learned_addr learned my addr 172.21.15.81:0/4264543636 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f539ea00640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a81546f0 0x7f53a814dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7400640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a80732f0 0x7f53a81f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 -- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a80732f0 msgr2=0x7f53a81f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a80732f0 0x7f53a81f6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 -- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a81546f0 msgr2=0x7f53a814dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a81546f0 0x7f53a814dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:53.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 -- 172.21.15.81:0/4264543636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a81f7030 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53a7e00640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f53a001bd20 tx=0x7f53a001beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00b4040 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a00c1070 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0009450 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a81f6610 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a8152970 con 0x7f53a815a510 2024-05-27T06:47:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a806a560 con 0x7f53a815a510 2024-05-27T06:47:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.846+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f53a00ba6b0 con 0x7f53a815a510 2024-05-27T06:47:53.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.850+0000 7f539fe00640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 0x7f5384071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:53.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.850+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f53a012dc70 con 0x7f53a815a510 2024-05-27T06:47:53.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.850+0000 7f53a7400640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 0x7f5384071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:53.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.850+0000 7f53a7400640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 0x7f5384071910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5398006e90 tx=0x7f5398008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:53.854+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f53a0018c70 con 0x7f53a815a510 2024-05-27T06:47:54.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.042+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53a8153c20 con 0x7f53a815a510 2024-05-27T06:47:54.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.042+0000 7f539fe00640 1 -- 172.21.15.81:0/4264543636 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6079 (secure 0 0 0) 0x7f53a00fa430 con 0x7f53a815a510 2024-05-27T06:47:54.039 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:54.039 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4235,4287,4281,4285,4302]},"id":1}]} 2024-05-27T06:47:54.039 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 22 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 msgr2=0x7f5384071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 0x7f5384071910 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5398006e90 tx=0x7f5398008040 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 msgr2=0x7f53a814fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f53a001bd20 tx=0x7f53a001beb0 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 shutdown_connections 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f538406f400 0x7f5384071910 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a815a510 0x7f53a814fce0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53a81546f0 0x7f53a814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 --2- 172.21.15.81:0/4264543636 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f53a80732f0 0x7f53a81f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 >> 172.21.15.81:0/4264543636 conn(0x7f53a8070700 msgr2=0x7f53a81509a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 shutdown_connections 2024-05-27T06:47:54.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:54.046+0000 7f53ae600640 1 -- 172.21.15.81:0/4264543636 wait complete. 2024-05-27T06:47:54.060 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-05-27T06:47:54.320 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-05-27T06:47:54.343 INFO:tasks.workunit.client.0.smithi081.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-05-27T06:47:54.566 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo passed: files are identical 2024-05-27T06:47:54.566 INFO:tasks.workunit.client.0.smithi081.stdout:passed: files are identical 2024-05-27T06:47:54.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir foo/.snap/barsnap 2024-05-27T06:47:54.599 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:47:54.600 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:47:54.600 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:47:54.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:47:54.658 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-05-27T06:47:54.659 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-05-27T06:47:54.681 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 1 2024-05-27T06:47:54.684 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 2 2024-05-27T06:47:54.686 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 3 2024-05-27T06:47:54.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ '[' -e file4 ']' 2024-05-27T06:47:54.689 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir .snap/snap1 2024-05-27T06:47:54.702 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 4 2024-05-27T06:47:54.704 INFO:tasks.workunit.client.0.smithi081.stderr:++ ls 2024-05-27T06:47:54.711 INFO:tasks.workunit.client.0.smithi081.stderr:+ now='file1 2024-05-27T06:47:54.711 INFO:tasks.workunit.client.0.smithi081.stderr:file2 2024-05-27T06:47:54.711 INFO:tasks.workunit.client.0.smithi081.stderr:file3 2024-05-27T06:47:54.711 INFO:tasks.workunit.client.0.smithi081.stderr:file4' 2024-05-27T06:47:54.712 INFO:tasks.workunit.client.0.smithi081.stderr:++ ls .snap/snap1 2024-05-27T06:47:54.720 INFO:tasks.workunit.client.0.smithi081.stderr:+ then='file1 2024-05-27T06:47:54.720 INFO:tasks.workunit.client.0.smithi081.stderr:file2 2024-05-27T06:47:54.720 INFO:tasks.workunit.client.0.smithi081.stderr:file3' 2024-05-27T06:47:54.721 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir .snap/snap1 2024-05-27T06:47:54.729 INFO:tasks.workunit.client.0.smithi081.stderr:+ '[' 'file1 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:file2 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:file3 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:file4' = 'file1 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:file2 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:file3' ']' 2024-05-27T06:47:54.730 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 1 2024-05-27T06:47:54.736 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 2 2024-05-27T06:47:54.747 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 3 2024-05-27T06:47:54.755 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir .snap/snap1 2024-05-27T06:47:54.766 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 4 2024-05-27T06:47:54.773 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir .snap/snap1 2024-05-27T06:47:54.791 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm file1 file2 file3 file4 2024-05-27T06:47:54.806 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:47:54.807 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:47:54.807 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:47:54.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:47:54.883 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-05-27T06:47:54.883 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-05-27T06:47:54.945 INFO:tasks.workunit.client.0.smithi081.stdout:Create dir 100 to 199 ... 2024-05-27T06:47:54.947 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 100 2024-05-27T06:47:54.978 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 101 2024-05-27T06:47:55.008 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 102 2024-05-27T06:47:55.040 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 103 2024-05-27T06:47:55.072 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 104 2024-05-27T06:47:55.103 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 105 2024-05-27T06:47:55.134 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 106 2024-05-27T06:47:55.164 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 107 2024-05-27T06:47:55.195 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 108 2024-05-27T06:47:55.225 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 109 2024-05-27T06:47:55.256 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 110 2024-05-27T06:47:55.286 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 111 2024-05-27T06:47:55.318 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 112 2024-05-27T06:47:55.347 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 113 2024-05-27T06:47:55.378 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 114 2024-05-27T06:47:55.408 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 115 2024-05-27T06:47:55.438 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 116 2024-05-27T06:47:55.469 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 117 2024-05-27T06:47:55.651 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 118 2024-05-27T06:47:56.033 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 119 2024-05-27T06:47:56.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:56.065 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 120 2024-05-27T06:47:56.098 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 121 2024-05-27T06:47:56.129 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 122 2024-05-27T06:47:56.163 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 123 2024-05-27T06:47:56.194 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 124 2024-05-27T06:47:56.226 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 125 2024-05-27T06:47:56.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 Processor -- start 2024-05-27T06:47:56.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 -- start start 2024-05-27T06:47:56.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00ca770 0x7ff3b00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b0154200 0x7ff3b0154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b00c78b0 con 0x7ff3b00cbd00 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b0154b50 con 0x7ff3b0154200 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.242+0000 7ff3b6400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00c6ee0 con 0x7ff3b00ca770 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b0154200 0x7ff3b0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3af400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00ca770 0x7ff3b00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b0154200 0x7ff3b0154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36948/0 (socket says 172.21.15.81:36948) 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3aea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff3b00cbd00 0x7ff3b00cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34592/0 (socket says 172.21.15.81:34592) 2024-05-27T06:47:56.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 -- 172.21.15.81:0/2837274872 learned_addr learned my addr 172.21.15.81:0/2837274872 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 -- 172.21.15.81:0/2837274872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00ca770 msgr2=0x7ff3b00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 --2- 172.21.15.81:0/2837274872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00ca770 0x7ff3b00cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 -- 172.21.15.81:0/2837274872 >> v1:172.21.15.81:6789/0 conn(0x7ff3b00cbd00 legacy=0x7ff3b00cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3ae000640 1 -- 172.21.15.81:0/2837274872 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2759270955 0 0) 0x7ff3b00c78b0 con 0x7ff3b00cbd00 2024-05-27T06:47:56.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.246+0000 7ff3afe00640 1 -- 172.21.15.81:0/2837274872 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b00c71c0 con 0x7ff3b0154200 2024-05-27T06:47:56.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3afe00640 1 --2- 172.21.15.81:0/2837274872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b0154200 0x7ff3b0154610 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009e50 tx=0x7ff3a801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949b7cd305d471f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:56.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3ae000640 1 -- 172.21.15.81:0/2837274872 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80a5810 con 0x7ff3b0154200 2024-05-27T06:47:56.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3ae000640 1 -- 172.21.15.81:0/2837274872 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a80a59b0 con 0x7ff3b0154200 2024-05-27T06:47:56.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3ae000640 1 -- 172.21.15.81:0/2837274872 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80a5d80 con 0x7ff3b0154200 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 -- 172.21.15.81:0/2837274872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b0154200 msgr2=0x7ff3b0154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2837274872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b0154200 0x7ff3b0154610 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8009e50 tx=0x7ff3a801ca30 comp rx=0 tx=0).stop 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 -- 172.21.15.81:0/2837274872 shutdown_connections 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2837274872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b0154200 0x7ff3b0154610 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2837274872 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00ca770 0x7ff3b00cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 -- 172.21.15.81:0/2837274872 >> 172.21.15.81:0/2837274872 conn(0x7ff3b0068ae0 msgr2=0x7ff3b0068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 -- 172.21.15.81:0/2837274872 shutdown_connections 2024-05-27T06:47:56.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.250+0000 7ff3b6400640 1 -- 172.21.15.81:0/2837274872 wait complete. 2024-05-27T06:47:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 Processor -- start 2024-05-27T06:47:56.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- start start 2024-05-27T06:47:56.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00cbd00 0x7ff3b0156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b0154200 0x7ff3b0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00d0640 con 0x7ff3b00ca770 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00d04c0 con 0x7ff3b0154200 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b0167590 con 0x7ff3b00cbd00 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36956/0 (socket says 172.21.15.81:36956) 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 -- 172.21.15.81:0/2652050290 learned_addr learned my addr 172.21.15.81:0/2652050290 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3aea00640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00cbd00 0x7ff3b0156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3afe00640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b0154200 0x7ff3b0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 -- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00cbd00 msgr2=0x7ff3b0156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00cbd00 0x7ff3b0156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 -- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b0154200 msgr2=0x7ff3b0159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b0154200 0x7ff3b0159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 -- 172.21.15.81:0/2652050290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b01f57c0 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3af400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000e870 tx=0x7ff3a000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0017070 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a000c940 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a000ccf0 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b006d940 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.254+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b0157230 con 0x7ff3b00ca770 2024-05-27T06:47:56.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.258+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b006a560 con 0x7ff3b00ca770 2024-05-27T06:47:56.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.258+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff3a001a070 con 0x7ff3b00ca770 2024-05-27T06:47:56.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.258+0000 7ff3a7400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 0x7ff390071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:56.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.262+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff3a008f5b0 con 0x7ff3b00ca770 2024-05-27T06:47:56.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.262+0000 7ff3aea00640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 0x7ff390071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:56.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.262+0000 7ff3aea00640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 0x7ff390071910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff3b01f68f0 tx=0x7ff39c007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:56.257 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 126 2024-05-27T06:47:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.266+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff3a005bd70 con 0x7ff3b00ca770 2024-05-27T06:47:56.286 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 127 2024-05-27T06:47:56.396 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 128 2024-05-27T06:47:56.429 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 129 2024-05-27T06:47:56.462 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 130 2024-05-27T06:47:56.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.490+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3b0158040 con 0x7ff3b00ca770 2024-05-27T06:47:56.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3a7400640 1 -- 172.21.15.81:0/2652050290 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7ff3a0060070 con 0x7ff3b00ca770 2024-05-27T06:47:56.488 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:56.488 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:47:56.489 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:47:56.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 msgr2=0x7ff390071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 0x7ff390071910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff3b01f68f0 tx=0x7ff39c007df0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 msgr2=0x7ff3b0157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:56.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000e870 tx=0x7ff3a000ec70 comp rx=0 tx=0).stop 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 shutdown_connections 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff3b00cbd00 0x7ff3b0156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff39006f400 0x7ff390071910 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3b0154200 0x7ff3b0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 --2- 172.21.15.81:0/2652050290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00ca770 0x7ff3b0157a30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 >> 172.21.15.81:0/2652050290 conn(0x7ff3b0068ae0 msgr2=0x7ff3b015c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 shutdown_connections 2024-05-27T06:47:56.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:56.494+0000 7ff3b6400640 1 -- 172.21.15.81:0/2652050290 wait complete. 2024-05-27T06:47:56.492 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 131 2024-05-27T06:47:56.514 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-05-27T06:47:56.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2024-05-27T06:47:56.523 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 132 2024-05-27T06:47:56.552 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 133 2024-05-27T06:47:56.582 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 134 2024-05-27T06:47:56.896 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 135 2024-05-27T06:47:56.925 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 136 2024-05-27T06:47:56.954 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 137 2024-05-27T06:47:56.984 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 138 2024-05-27T06:47:57.017 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 139 2024-05-27T06:47:57.045 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 140 2024-05-27T06:47:57.074 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 141 2024-05-27T06:47:57.104 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 142 2024-05-27T06:47:57.132 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 143 2024-05-27T06:47:57.163 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 144 2024-05-27T06:47:57.194 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 145 2024-05-27T06:47:57.225 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 146 2024-05-27T06:47:57.255 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 147 2024-05-27T06:47:57.283 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 148 2024-05-27T06:47:57.313 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 149 2024-05-27T06:47:57.342 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 150 2024-05-27T06:47:57.373 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 151 2024-05-27T06:47:57.405 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 152 2024-05-27T06:47:57.439 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 153 2024-05-27T06:47:57.472 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 154 2024-05-27T06:47:57.505 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 155 2024-05-27T06:47:57.534 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 156 2024-05-27T06:47:57.746 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 157 2024-05-27T06:47:57.778 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 158 2024-05-27T06:47:57.808 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 159 2024-05-27T06:47:57.838 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 160 2024-05-27T06:47:58.516 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:58.560 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 161 2024-05-27T06:47:58.592 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 162 2024-05-27T06:47:58.624 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 163 2024-05-27T06:47:58.659 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 164 2024-05-27T06:47:58.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.690+0000 7f2b8e400640 1 Processor -- start 2024-05-27T06:47:58.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 -- start start 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2b880c9850 0x7f2b880ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2b88071f20 0x7f2b88072330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b8815c8d0 con 0x7f2b88160040 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f2b8815d7a0 con 0x7f2b88071f20 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8e400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f2b8815cb90 con 0x7f2b880c9850 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2b88071f20 0x7f2b88072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2b88071f20 0x7f2b88072330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36962/0 (socket says 172.21.15.81:36962) 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 -- 172.21.15.81:0/3083946772 learned_addr learned my addr 172.21.15.81:0/3083946772 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b8d000640 1 --2- 172.21.15.81:0/3083946772 >> v2:172.21.15.106:3300/0 conn(0x7f2b880c9850 0x7f2b880ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 -- 172.21.15.81:0/3083946772 >> v2:172.21.15.106:3300/0 conn(0x7f2b880c9850 msgr2=0x7f2b880ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:58.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 --2- 172.21.15.81:0/3083946772 >> v2:172.21.15.106:3300/0 conn(0x7f2b880c9850 0x7f2b880ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 -- 172.21.15.81:0/3083946772 >> v1:172.21.15.106:6789/0 conn(0x7f2b88160040 legacy=0x7f2b88160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:47:58.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.694+0000 7f2b87e00640 1 -- 172.21.15.81:0/3083946772 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8815ce70 con 0x7f2b88071f20 2024-05-27T06:47:58.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b87e00640 1 --2- 172.21.15.81:0/3083946772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88071f20 0x7f2b88072330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009870 tx=0x7f2b7401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bb2c8dcfe94c0e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b87400640 1 -- 172.21.15.81:0/3083946772 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740b0070 con 0x7f2b88071f20 2024-05-27T06:47:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b87400640 1 -- 172.21.15.81:0/3083946772 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b740ab600 con 0x7f2b88071f20 2024-05-27T06:47:58.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b87400640 1 -- 172.21.15.81:0/3083946772 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740ab9d0 con 0x7f2b88071f20 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 -- 172.21.15.81:0/3083946772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88071f20 msgr2=0x7f2b88072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 --2- 172.21.15.81:0/3083946772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88071f20 0x7f2b88072330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2b74009870 tx=0x7f2b7401c520 comp rx=0 tx=0).stop 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 -- 172.21.15.81:0/3083946772 shutdown_connections 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 --2- 172.21.15.81:0/3083946772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88071f20 0x7f2b88072330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 --2- 172.21.15.81:0/3083946772 >> v2:172.21.15.106:3300/0 conn(0x7f2b880c9850 0x7f2b880ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 -- 172.21.15.81:0/3083946772 >> 172.21.15.81:0/3083946772 conn(0x7f2b88068d90 msgr2=0x7f2b880cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 -- 172.21.15.81:0/3083946772 shutdown_connections 2024-05-27T06:47:58.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.698+0000 7f2b8e400640 1 -- 172.21.15.81:0/3083946772 wait complete. 2024-05-27T06:47:58.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 Processor -- start 2024-05-27T06:47:58.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 -- start start 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 0x7f2b881f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880ca9a0 con 0x7f2b88160040 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880ca820 con 0x7f2b88071f20 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8e400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2b880c9c60 con 0x7f2b880c9850 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8d000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8d000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:50562/0 (socket says 172.21.15.81:50562) 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36978/0 (socket says 172.21.15.81:36978) 2024-05-27T06:47:58.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 0x7f2b881f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8d000640 1 -- 172.21.15.81:0/2147817423 learned_addr learned my addr 172.21.15.81:0/2147817423 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 -- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 msgr2=0x7f2b8814d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 -- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 msgr2=0x7f2b88149200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 -- 172.21.15.81:0/2147817423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8814f470 con 0x7f2b880c9850 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8d000640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:47:58.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b8da00640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.702+0000 7f2b87e00640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 0x7f2b881f5d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b740ab000 tx=0x7f2b740192c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740ba070 con 0x7f2b880c9850 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b740b4420 con 0x7f2b880c9850 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b881f9410 con 0x7f2b880c9850 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740b0040 con 0x7f2b880c9850 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b881f99d0 con 0x7f2b880c9850 2024-05-27T06:47:58.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b88050870 con 0x7f2b880c9850 2024-05-27T06:47:58.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2b740bd070 con 0x7f2b880c9850 2024-05-27T06:47:58.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 0x7f2b68071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:58.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.706+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5967+0+0 (secure 0 0 0) 0x7f2b74128330 con 0x7f2b880c9850 2024-05-27T06:47:58.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.710+0000 7f2b8d000640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 0x7f2b68071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.710+0000 7f2b8d000640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 0x7f2b68071910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c007e60 tx=0x7f2b7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.710+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2b74016110 con 0x7f2b880c9850 2024-05-27T06:47:58.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.898+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b8806de60 con 0x7f2b880c9850 2024-05-27T06:47:58.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b84c00640 1 -- 172.21.15.81:0/2147817423 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7f2b740fd210 con 0x7f2b880c9850 2024-05-27T06:47:58.896 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:58.896 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:47:58.897 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:47:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 msgr2=0x7f2b68071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 0x7f2b68071910 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c007e60 tx=0x7f2b7c008040 comp rx=0 tx=0).stop 2024-05-27T06:47:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 msgr2=0x7f2b881f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 0x7f2b881f5d00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b740ab000 tx=0x7f2b740192c0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 shutdown_connections 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b880c9850 0x7f2b881f5d00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b6806f400 0x7f2b68071910 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b88160040 0x7f2b88149200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 --2- 172.21.15.81:0/2147817423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b88071f20 0x7f2b8814d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 >> 172.21.15.81:0/2147817423 conn(0x7f2b88068d90 msgr2=0x7f2b880cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 shutdown_connections 2024-05-27T06:47:58.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:58.902+0000 7f2b8e400640 1 -- 172.21.15.81:0/2147817423 wait complete. 2024-05-27T06:47:58.918 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=4) 2024-05-27T06:47:58.918 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-05-27T06:47:58.918 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:47:58.977 INFO:tasks.ceph.mds.e:Stopped 2024-05-27T06:47:58.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:47:59.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 Processor -- start 2024-05-27T06:47:59.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 -- start start 2024-05-27T06:47:59.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc6b40c5f70 0x7fc6b4165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:59.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b4072840 con 0x7fc6b40c5b60 2024-05-27T06:47:59.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b40c5270 con 0x7fc6b4072430 2024-05-27T06:47:59.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.170+0000 7fc6bb800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b40c5040 con 0x7fc6b40c5f70 2024-05-27T06:47:59.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc6b40c5b60 0x7fc6b4161cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34596/0 (socket says 172.21.15.81:34596) 2024-05-27T06:47:59.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3e00640 1 -- 172.21.15.81:0/1837325298 learned_addr learned my addr 172.21.15.81:0/1837325298 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:47:59.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b9800640 1 --2- 172.21.15.81:0/1837325298 >> v2:172.21.15.106:3300/0 conn(0x7fc6b40c5f70 0x7fc6b4165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087158537 0 0) 0x7fc6b4072840 con 0x7fc6b40c5b60 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc698003530 con 0x7fc6b40c5b60 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537566814 0 0) 0x7fc6b40c5270 con 0x7fc6b4072430 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b4072840 con 0x7fc6b4072430 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b9800640 1 -- 172.21.15.81:0/1837325298 >> v1:172.21.15.106:6789/0 conn(0x7fc6b4072430 legacy=0x7fc6b4159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b9800640 1 -- 172.21.15.81:0/1837325298 >> v1:172.21.15.81:6789/0 conn(0x7fc6b40c5b60 legacy=0x7fc6b4161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b9800640 1 -- 172.21.15.81:0/1837325298 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b40c7810 con 0x7fc6b40c5f70 2024-05-27T06:47:59.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.174+0000 7fc6b9800640 1 --2- 172.21.15.81:0/1837325298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b40c5f70 0x7fc6b4165400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097c0 tx=0x7fc6ac01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8256bf0e1115756 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:59.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0b0070 con 0x7fc6b40c5f70 2024-05-27T06:47:59.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac0ab600 con 0x7fc6b40c5f70 2024-05-27T06:47:59.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6b3400640 1 -- 172.21.15.81:0/1837325298 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0ab9d0 con 0x7fc6b40c5f70 2024-05-27T06:47:59.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- 172.21.15.81:0/1837325298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b40c5f70 msgr2=0x7fc6b4165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 --2- 172.21.15.81:0/1837325298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b40c5f70 0x7fc6b4165400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097c0 tx=0x7fc6ac01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- 172.21.15.81:0/1837325298 shutdown_connections 2024-05-27T06:47:59.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 --2- 172.21.15.81:0/1837325298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b40c5f70 0x7fc6b4165400 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- 172.21.15.81:0/1837325298 >> 172.21.15.81:0/1837325298 conn(0x7fc6b4070700 msgr2=0x7fc6b40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:59.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- 172.21.15.81:0/1837325298 shutdown_connections 2024-05-27T06:47:59.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- 172.21.15.81:0/1837325298 wait complete. 2024-05-27T06:47:59.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 Processor -- start 2024-05-27T06:47:59.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.178+0000 7fc6bb800640 1 -- start start 2024-05-27T06:47:59.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 0x7fc6b41f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:59.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6b40c5b60 0x7fc6b4151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:59.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:59.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b40c7dd0 con 0x7fc6b40c5f70 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b40c75c0 con 0x7fc6b4072430 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b40ce6f0 con 0x7fc6b40c5b60 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36982/0 (socket says 172.21.15.81:36982) 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 -- 172.21.15.81:0/416399074 learned_addr learned my addr 172.21.15.81:0/416399074 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b3e00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6b40c5b60 0x7fc6b4151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b8e00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 0x7fc6b41f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 -- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6b40c5b60 msgr2=0x7fc6b4151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6b40c5b60 0x7fc6b4151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 -- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 msgr2=0x7fc6b41f94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 0x7fc6b41f94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 -- 172.21.15.81:0/416399074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b41f9a00 con 0x7fc6b40c5f70 2024-05-27T06:47:59.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b8e00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 0x7fc6b41f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b9800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097c0 tx=0x7fc6ac01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0bc070 con 0x7fc6b40c5f70 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac0b1040 con 0x7fc6b40c5f70 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0b0050 con 0x7fc6b40c5f70 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b41f5f70 con 0x7fc6b40c5f70 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b41f6530 con 0x7fc6b40c5f70 2024-05-27T06:47:59.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.182+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b406a560 con 0x7fc6b40c5f70 2024-05-27T06:47:59.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.186+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc6ac0bf070 con 0x7fc6b40c5f70 2024-05-27T06:47:59.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.186+0000 7fc6b0c00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 0x7fc694071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:47:59.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.186+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5967+0+0 (secure 0 0 0) 0x7fc6ac128cd0 con 0x7fc6b40c5f70 2024-05-27T06:47:59.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.186+0000 7fc6b8e00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 0x7fc694071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:47:59.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.186+0000 7fc6b8e00640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 0x7fc694071910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40014b0 tx=0x7fc6a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:47:59.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.190+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc6ac016090 con 0x7fc6b40c5f70 2024-05-27T06:47:59.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6b40c5940 con 0x7fc6b40c5f70 2024-05-27T06:47:59.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6b0c00640 1 -- 172.21.15.81:0/416399074 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7fc6ac0f54f0 con 0x7fc6b40c5f70 2024-05-27T06:47:59.377 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:47:59.377 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:47:59.377 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 msgr2=0x7fc694071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 0x7fc694071910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40014b0 tx=0x7fc6a4009040 comp rx=0 tx=0).stop 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 msgr2=0x7fc6b41461d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097c0 tx=0x7fc6ac01ace0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 shutdown_connections 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc6b40c5b60 0x7fc6b4151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc69406f400 0x7fc694071910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c5f70 0x7fc6b41461d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 --2- 172.21.15.81:0/416399074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6b4072430 0x7fc6b41f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:47:59.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 >> 172.21.15.81:0/416399074 conn(0x7fc6b4070700 msgr2=0x7fc6b40c8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:47:59.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 shutdown_connections 2024-05-27T06:47:59.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:47:59.382+0000 7fc6bb800640 1 -- 172.21.15.81:0/416399074 wait complete. 2024-05-27T06:47:59.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:47:59.556 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 165 2024-05-27T06:47:59.586 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 166 2024-05-27T06:47:59.616 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 167 2024-05-27T06:47:59.648 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 168 2024-05-27T06:47:59.678 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 169 2024-05-27T06:47:59.707 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 170 2024-05-27T06:47:59.737 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 171 2024-05-27T06:47:59.766 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 172 2024-05-27T06:47:59.796 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 173 2024-05-27T06:47:59.827 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 174 2024-05-27T06:47:59.859 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 175 2024-05-27T06:47:59.888 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 176 2024-05-27T06:47:59.919 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 177 2024-05-27T06:47:59.950 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 178 2024-05-27T06:47:59.980 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 179 2024-05-27T06:48:00.009 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 180 2024-05-27T06:48:00.039 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 181 2024-05-27T06:48:00.069 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 182 2024-05-27T06:48:00.304 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 183 2024-05-27T06:48:00.334 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 184 2024-05-27T06:48:00.365 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 185 2024-05-27T06:48:00.395 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 186 2024-05-27T06:48:01.190 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 187 2024-05-27T06:48:01.222 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 188 2024-05-27T06:48:01.254 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 189 2024-05-27T06:48:01.284 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 190 2024-05-27T06:48:01.315 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 191 2024-05-27T06:48:01.347 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 192 2024-05-27T06:48:01.377 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 193 2024-05-27T06:48:01.399 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:01.406 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 194 2024-05-27T06:48:01.437 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 195 2024-05-27T06:48:01.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.562+0000 7efc5ca00640 1 Processor -- start 2024-05-27T06:48:01.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.562+0000 7efc5ca00640 1 -- start start 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc5ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc5ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc580c9f80 con 0x7efc580c8880 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc5ca00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc580c52f0 con 0x7efc5815cb80 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc5ca00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efc580c58b0 con 0x7efc5806c790 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc56800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efc580c8880 0x7efc580c8c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34600/0 (socket says 172.21.15.81:34600) 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc56800640 1 -- 172.21.15.81:0/2264214691 learned_addr learned my addr 172.21.15.81:0/2264214691 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:01.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc4fe00640 1 --2- 172.21.15.81:0/2264214691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:01.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423330312 0 0) 0x7efc580c9f80 con 0x7efc580c8880 2024-05-27T06:48:01.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc3c003530 con 0x7efc580c8880 2024-05-27T06:48:01.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565815878 0 0) 0x7efc580c52f0 con 0x7efc5815cb80 2024-05-27T06:48:01.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc580c9f80 con 0x7efc5815cb80 2024-05-27T06:48:01.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc4fe00640 1 -- 172.21.15.81:0/2264214691 >> v1:172.21.15.106:6789/0 conn(0x7efc5815cb80 legacy=0x7efc58152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc4fe00640 1 -- 172.21.15.81:0/2264214691 >> v1:172.21.15.81:6789/0 conn(0x7efc580c8880 legacy=0x7efc580c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.566+0000 7efc4fe00640 1 -- 172.21.15.81:0/2264214691 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc580c55d0 con 0x7efc5806c790 2024-05-27T06:48:01.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc4fe00640 1 --2- 172.21.15.81:0/2264214691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5806cba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efc4000a990 tx=0x7efc4001d8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=529644e3c4b1f138 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:01.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4000b040 con 0x7efc5806c790 2024-05-27T06:48:01.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc400a7710 con 0x7efc5806c790 2024-05-27T06:48:01.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc55400640 1 -- 172.21.15.81:0/2264214691 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc400a7ae0 con 0x7efc5806c790 2024-05-27T06:48:01.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 -- 172.21.15.81:0/2264214691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 msgr2=0x7efc5806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2264214691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5806cba0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efc4000a990 tx=0x7efc4001d8b0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 -- 172.21.15.81:0/2264214691 shutdown_connections 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2264214691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5806cba0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 -- 172.21.15.81:0/2264214691 >> 172.21.15.81:0/2264214691 conn(0x7efc58070700 msgr2=0x7efc580cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 -- 172.21.15.81:0/2264214691 shutdown_connections 2024-05-27T06:48:01.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.570+0000 7efc5ca00640 1 -- 172.21.15.81:0/2264214691 wait complete. 2024-05-27T06:48:01.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 Processor -- start 2024-05-27T06:48:01.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- start start 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5814a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc580c8880 0x7efc581f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc580c9d10 con 0x7efc5815cb80 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efc580c83f0 con 0x7efc580c8880 2024-05-27T06:48:01.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efc58166710 con 0x7efc5806c790 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc55e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5814a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc4fe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc580c8880 0x7efc581f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36996/0 (socket says 172.21.15.81:36996) 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 -- 172.21.15.81:0/2649601102 learned_addr learned my addr 172.21.15.81:0/2649601102 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 -- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 msgr2=0x7efc5814a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5814a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 -- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc580c8880 msgr2=0x7efc581f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc580c8880 0x7efc581f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 -- 172.21.15.81:0/2649601102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581f7770 con 0x7efc5815cb80 2024-05-27T06:48:01.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc56800640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efc5000ca70 tx=0x7efc5000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5000ab70 con 0x7efc5815cb80 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc5000ad10 con 0x7efc5815cb80 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50012640 con 0x7efc5815cb80 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc580c66d0 con 0x7efc5815cb80 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.574+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc58162ac0 con 0x7efc5815cb80 2024-05-27T06:48:01.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc5806a560 con 0x7efc5815cb80 2024-05-27T06:48:01.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efc500127e0 con 0x7efc5815cb80 2024-05-27T06:48:01.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc4e000640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 0x7efc34071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:01.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5967+0+0 (secure 0 0 0) 0x7efc5008f8d0 con 0x7efc5815cb80 2024-05-27T06:48:01.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc55e00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 0x7efc34071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:01.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.578+0000 7efc55e00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 0x7efc34071940 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc48000c10 tx=0x7efc48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:01.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.582+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7efc5005bfd0 con 0x7efc5815cb80 2024-05-27T06:48:01.775 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 196 2024-05-27T06:48:01.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.794+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc5814a9c0 con 0x7efc5815cb80 2024-05-27T06:48:01.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.794+0000 7efc4e000640 1 -- 172.21.15.81:0/2649601102 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7efc500602d0 con 0x7efc5815cb80 2024-05-27T06:48:01.790 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:01.790 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:48:01.790 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 msgr2=0x7efc34071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 0x7efc34071940 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efc48000c10 tx=0x7efc48009040 comp rx=0 tx=0).stop 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 msgr2=0x7efc58149080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efc5000ca70 tx=0x7efc5000cf30 comp rx=0 tx=0).stop 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 shutdown_connections 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efc3406f430 0x7efc34071940 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc5815cb80 0x7efc58149080 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc580c8880 0x7efc581f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 --2- 172.21.15.81:0/2649601102 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efc5806c790 0x7efc5814a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:01.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 >> 172.21.15.81:0/2649601102 conn(0x7efc58070700 msgr2=0x7efc580c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:01.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 shutdown_connections 2024-05-27T06:48:01.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:01.798+0000 7efc5ca00640 1 -- 172.21.15.81:0/2649601102 wait complete. 2024-05-27T06:48:01.812 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:48:01.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:48:01.858 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:48:01.900 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 197 2024-05-27T06:48:01.934 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 198 2024-05-27T06:48:01.961 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 199 2024-05-27T06:48:01.993 INFO:tasks.workunit.client.0.smithi081.stdout:Take first snapshot .snap/test1 2024-05-27T06:48:03.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:03.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 Processor -- start 2024-05-27T06:48:03.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 -- start start 2024-05-27T06:48:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 0x7efd880cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd880c78b0 con 0x7efd880cbd00 2024-05-27T06:48:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd880c6ee0 con 0x7efd88154200 2024-05-27T06:48:03.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd8e000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd880c71c0 con 0x7efd880ca770 2024-05-27T06:48:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd86a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efd880cbd00 0x7efd880cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34606/0 (socket says 172.21.15.81:34606) 2024-05-27T06:48:03.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd86a00640 1 -- 172.21.15.81:0/920083009 learned_addr learned my addr 172.21.15.81:0/920083009 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd87400640 1 --2- 172.21.15.81:0/920083009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 0x7efd880cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967139390 0 0) 0x7efd880c78b0 con 0x7efd880cbd00 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.978+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd6c003530 con 0x7efd880cbd00 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341279022 0 0) 0x7efd880c6ee0 con 0x7efd88154200 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd880c78b0 con 0x7efd88154200 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd87400640 1 -- 172.21.15.81:0/920083009 >> v1:172.21.15.106:6789/0 conn(0x7efd88154200 legacy=0x7efd88164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd87400640 1 -- 172.21.15.81:0/920083009 >> v1:172.21.15.81:6789/0 conn(0x7efd880cbd00 legacy=0x7efd880cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:03.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd87400640 1 -- 172.21.15.81:0/920083009 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd880c7b70 con 0x7efd880ca770 2024-05-27T06:48:03.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd87400640 1 --2- 172.21.15.81:0/920083009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 0x7efd880cb7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efd780097f0 tx=0x7efd7801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f848a74ef2a46cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:03.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780ab5f0 con 0x7efd880ca770 2024-05-27T06:48:03.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd780b0070 con 0x7efd880ca770 2024-05-27T06:48:03.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.982+0000 7efd86000640 1 -- 172.21.15.81:0/920083009 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780ab990 con 0x7efd880ca770 2024-05-27T06:48:03.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- 172.21.15.81:0/920083009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 msgr2=0x7efd880cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 --2- 172.21.15.81:0/920083009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 0x7efd880cb7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efd780097f0 tx=0x7efd7801c4e0 comp rx=0 tx=0).stop 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- 172.21.15.81:0/920083009 shutdown_connections 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 --2- 172.21.15.81:0/920083009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd880ca770 0x7efd880cb7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- 172.21.15.81:0/920083009 >> 172.21.15.81:0/920083009 conn(0x7efd88068ae0 msgr2=0x7efd88068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- 172.21.15.81:0/920083009 shutdown_connections 2024-05-27T06:48:03.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- 172.21.15.81:0/920083009 wait complete. 2024-05-27T06:48:03.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 Processor -- start 2024-05-27T06:48:03.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- start start 2024-05-27T06:48:03.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd880ca770 0x7efd88154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88154200 0x7efd881595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd880d0640 con 0x7efd880cbd00 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd880d04c0 con 0x7efd880ca770 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.986+0000 7efd8e000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd88168cb0 con 0x7efd88154200 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37002/0 (socket says 172.21.15.81:37002) 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 -- 172.21.15.81:0/3615490691 learned_addr learned my addr 172.21.15.81:0/3615490691 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd87400640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd880ca770 0x7efd88154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:03.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd87e00640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88154200 0x7efd881595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:03.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 -- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88154200 msgr2=0x7efd881595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88154200 0x7efd881595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 -- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd880ca770 msgr2=0x7efd88154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd880ca770 0x7efd88154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 -- 172.21.15.81:0/3615490691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd88157a50 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd86a00640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c460 tx=0x7efd7000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70017070 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd70005cf0 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd881f71b0 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70007310 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd881f7770 con 0x7efd880cbd00 2024-05-27T06:48:03.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8806a560 con 0x7efd880cbd00 2024-05-27T06:48:03.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.990+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efd7001a070 con 0x7efd880cbd00 2024-05-27T06:48:03.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.994+0000 7efd7f400640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 0x7efd64071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:03.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.994+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5939+0+0 (secure 0 0 0) 0x7efd7008e630 con 0x7efd880cbd00 2024-05-27T06:48:03.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.994+0000 7efd87400640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 0x7efd64071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:03.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.994+0000 7efd87400640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 0x7efd64071910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd780af000 tx=0x7efd780a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:03.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:03.998+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7efd7005ae60 con 0x7efd880cbd00 2024-05-27T06:48:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.186+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd880727b0 con 0x7efd880cbd00 2024-05-27T06:48:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.186+0000 7efd7f400640 1 -- 172.21.15.81:0/3615490691 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7efd7005f160 con 0x7efd880cbd00 2024-05-27T06:48:04.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:04.183 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:48:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 msgr2=0x7efd64071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 0x7efd64071910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efd780af000 tx=0x7efd780a7000 comp rx=0 tx=0).stop 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 msgr2=0x7efd88157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c460 tx=0x7efd7000c920 comp rx=0 tx=0).stop 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 shutdown_connections 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88154200 0x7efd881595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6406f400 0x7efd64071910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd880cbd00 0x7efd88157510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 --2- 172.21.15.81:0/3615490691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd880ca770 0x7efd88154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 >> 172.21.15.81:0/3615490691 conn(0x7efd88068ae0 msgr2=0x7efd8815b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 shutdown_connections 2024-05-27T06:48:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:04.190+0000 7efd8e000640 1 -- 172.21.15.81:0/3615490691 wait complete. 2024-05-27T06:48:04.204 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:48:06.205 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:06.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 Processor -- start 2024-05-27T06:48:06.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 -- start start 2024-05-27T06:48:06.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61841549b0 0x7f618415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f6184072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6184073910 con 0x7f618415e150 2024-05-27T06:48:06.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f61840c9ba0 con 0x7f61841549b0 2024-05-27T06:48:06.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6188e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6184073380 con 0x7f6184072a30 2024-05-27T06:48:06.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61841549b0 0x7f618415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61841549b0 0x7f618415dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:54514/0 (socket says 172.21.15.81:54514) 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 -- 172.21.15.81:0/4061908394 learned_addr learned my addr 172.21.15.81:0/4061908394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183e00640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f6184072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 -- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 msgr2=0x7f6184072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f6184072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 -- 172.21.15.81:0/4061908394 >> v1:172.21.15.106:6789/0 conn(0x7f618415e150 legacy=0x7f61840c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183400640 1 -- 172.21.15.81:0/4061908394 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6184073630 con 0x7f61841549b0 2024-05-27T06:48:06.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.370+0000 7f6183e00640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f6184072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:06.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6183400640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61841549b0 0x7f618415dc10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f617401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55abc4fda97649e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:06.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6182000640 1 -- 172.21.15.81:0/4061908394 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61740b2070 con 0x7f61841549b0 2024-05-27T06:48:06.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6182000640 1 -- 172.21.15.81:0/4061908394 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61740a9070 con 0x7f61841549b0 2024-05-27T06:48:06.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6182000640 1 -- 172.21.15.81:0/4061908394 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61740ad880 con 0x7f61841549b0 2024-05-27T06:48:06.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 -- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61841549b0 msgr2=0x7f618415dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61841549b0 0x7f618415dc10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6174009870 tx=0x7f617401c560 comp rx=0 tx=0).stop 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 -- 172.21.15.81:0/4061908394 shutdown_connections 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f6184072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 --2- 172.21.15.81:0/4061908394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61841549b0 0x7f618415dc10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 -- 172.21.15.81:0/4061908394 >> 172.21.15.81:0/4061908394 conn(0x7f618406d180 msgr2=0x7f61840cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 -- 172.21.15.81:0/4061908394 shutdown_connections 2024-05-27T06:48:06.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.374+0000 7f6188e00640 1 -- 172.21.15.81:0/4061908394 wait complete. 2024-05-27T06:48:06.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 Processor -- start 2024-05-27T06:48:06.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 -- start start 2024-05-27T06:48:06.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f61840cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f618415e150 0x7f61841649e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61840c7590 con 0x7f61841549b0 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f61840c7410 con 0x7f618415e150 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6188e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f61841677c0 con 0x7f6184072a30 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42358/0 (socket says 172.21.15.81:42358) 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 -- 172.21.15.81:0/2790599488 learned_addr learned my addr 172.21.15.81:0/2790599488 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6183e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f618415e150 0x7f61841649e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6183400640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f61840cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 -- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 msgr2=0x7f61840cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f61840cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 -- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f618415e150 msgr2=0x7f61841649e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f618415e150 0x7f61841649e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 -- 172.21.15.81:0/2790599488 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618414a870 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.378+0000 7f6182a00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f617000c3f0 tx=0x7f617000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170017070 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700052d0 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170003820 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61840cd170 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61841f8f90 con 0x7f61841549b0 2024-05-27T06:48:06.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61840ccc00 con 0x7f61841549b0 2024-05-27T06:48:06.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f617001a070 con 0x7f61841549b0 2024-05-27T06:48:06.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 0x7f6164071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:06.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5939+0+0 (secure 0 0 0) 0x7f617008e500 con 0x7f61841549b0 2024-05-27T06:48:06.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.382+0000 7f6183400640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 0x7f6164071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:06.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.386+0000 7f6183400640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 0x7f6164071940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61740af000 tx=0x7f61740a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:06.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.386+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6184073910 con 0x7f61841549b0 2024-05-27T06:48:06.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6184050870 con 0x7f61841549b0 2024-05-27T06:48:06.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f617b400640 1 -- 172.21.15.81:0/2790599488 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7f6170005480 con 0x7f61841549b0 2024-05-27T06:48:06.573 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:06.574 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:48:06.574 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 msgr2=0x7f6164071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 0x7f6164071940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f61740af000 tx=0x7f61740a7000 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 msgr2=0x7f61841644a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.578+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f617000c3f0 tx=0x7f617000c7f0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 shutdown_connections 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f616406f430 0x7f6164071940 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f618415e150 0x7f61841649e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61841549b0 0x7f61841644a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 --2- 172.21.15.81:0/2790599488 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6184072a30 0x7f61840cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 >> 172.21.15.81:0/2790599488 conn(0x7f618406d180 msgr2=0x7f61840c6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:06.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 shutdown_connections 2024-05-27T06:48:06.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:06.582+0000 7f6188e00640 1 -- 172.21.15.81:0/2790599488 wait complete. 2024-05-27T06:48:06.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:48:08.596 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:08.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 Processor -- start 2024-05-27T06:48:08.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 -- start start 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 0x7fad70159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad70072840 con 0x7fad7015a040 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad700c5270 con 0x7fad70161cd0 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad78800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fad700c5040 con 0x7fad70072430 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fad7015a040 0x7fad700c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51036/0 (socket says 172.21.15.81:51036) 2024-05-27T06:48:08.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75400640 1 -- 172.21.15.81:0/2194668528 learned_addr learned my addr 172.21.15.81:0/2194668528 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:08.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75e00640 1 --2- 172.21.15.81:0/2194668528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 0x7fad70159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1126666714 0 0) 0x7fad70072840 con 0x7fad7015a040 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad58003530 con 0x7fad7015a040 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950293662 0 0) 0x7fad700c5270 con 0x7fad70161cd0 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad70072840 con 0x7fad70161cd0 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 782436364 0 0) 0x7fad58003530 con 0x7fad7015a040 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75e00640 1 -- 172.21.15.81:0/2194668528 >> v1:172.21.15.106:6789/0 conn(0x7fad70161cd0 legacy=0x7fad701640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75e00640 1 -- 172.21.15.81:0/2194668528 >> v1:172.21.15.81:6789/0 conn(0x7fad7015a040 legacy=0x7fad700c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.762+0000 7fad75e00640 1 -- 172.21.15.81:0/2194668528 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad700c7810 con 0x7fad70072430 2024-05-27T06:48:08.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad75e00640 1 --2- 172.21.15.81:0/2194668528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 0x7fad70159b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad640097f0 tx=0x7fad6401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1483c6db5cda24a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:08.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640b0070 con 0x7fad70072430 2024-05-27T06:48:08.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad640ab400 con 0x7fad70072430 2024-05-27T06:48:08.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad74a00640 1 -- 172.21.15.81:0/2194668528 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60001700 con 0x7fad70072430 2024-05-27T06:48:08.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 -- 172.21.15.81:0/2194668528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 msgr2=0x7fad70159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 --2- 172.21.15.81:0/2194668528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 0x7fad70159b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad640097f0 tx=0x7fad6401c400 comp rx=0 tx=0).stop 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 -- 172.21.15.81:0/2194668528 shutdown_connections 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 --2- 172.21.15.81:0/2194668528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad70072430 0x7fad70159b00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 -- 172.21.15.81:0/2194668528 >> 172.21.15.81:0/2194668528 conn(0x7fad70070700 msgr2=0x7fad700cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 -- 172.21.15.81:0/2194668528 shutdown_connections 2024-05-27T06:48:08.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.766+0000 7fad78800640 1 -- 172.21.15.81:0/2194668528 wait complete. 2024-05-27T06:48:08.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 Processor -- start 2024-05-27T06:48:08.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 -- start start 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 0x7fad701463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 0x7fad700cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fad700c7dd0 con 0x7fad70161cd0 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad700c75c0 con 0x7fad70072430 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad78800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fad700c5610 con 0x7fad7015a040 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:08.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42360/0 (socket says 172.21.15.81:42360) 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 -- 172.21.15.81:0/2027059941 learned_addr learned my addr 172.21.15.81:0/2027059941 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad75400640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 0x7fad700cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad75e00640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 0x7fad701463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 -- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 msgr2=0x7fad700cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 0x7fad700cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 -- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 msgr2=0x7fad701463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 0x7fad701463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad76800640 1 -- 172.21.15.81:0/2027059941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad701fa120 con 0x7fad70161cd0 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad75400640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 0x7fad700cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.770+0000 7fad75e00640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 0x7fad701463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:48:08.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad76800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00c700 tx=0x7fad6c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c003b80 con 0x7fad70161cd0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad6c003d20 con 0x7fad70161cd0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c00aa90 con 0x7fad70161cd0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad701fa3e0 con 0x7fad70161cd0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7015b530 con 0x7fad70161cd0 2024-05-27T06:48:08.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad7006a560 con 0x7fad70161cd0 2024-05-27T06:48:08.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fad6c00ac30 con 0x7fad70161cd0 2024-05-27T06:48:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 0x7fad54071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5939+0+0 (secure 0 0 0) 0x7fad6c08f2f0 con 0x7fad70161cd0 2024-05-27T06:48:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.774+0000 7fad75e00640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 0x7fad54071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:08.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.778+0000 7fad75e00640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 0x7fad54071940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad640ad000 tx=0x7fad64008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:08.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.782+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fad6c05c050 con 0x7fad70161cd0 2024-05-27T06:48:08.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.970+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad700ca640 con 0x7fad70161cd0 2024-05-27T06:48:08.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.970+0000 7fad6a000640 1 -- 172.21.15.81:0/2027059941 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7fad6c05b950 con 0x7fad70161cd0 2024-05-27T06:48:08.967 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:08.967 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:48:08.967 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 msgr2=0x7fad54071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 0x7fad54071940 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fad640ad000 tx=0x7fad64008040 comp rx=0 tx=0).stop 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 msgr2=0x7fad70140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00c700 tx=0x7fad6c00cbc0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 shutdown_connections 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fad7015a040 0x7fad700cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fad5406f430 0x7fad54071940 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad70161cd0 0x7fad70140fa0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 --2- 172.21.15.81:0/2027059941 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad70072430 0x7fad701463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:08.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 >> 172.21.15.81:0/2027059941 conn(0x7fad70070700 msgr2=0x7fad700c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:08.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 shutdown_connections 2024-05-27T06:48:08.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:08.974+0000 7fad78800640 1 -- 172.21.15.81:0/2027059941 wait complete. 2024-05-27T06:48:08.988 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:48:10.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:11.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 Processor -- start 2024-05-27T06:48:11.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 -- start start 2024-05-27T06:48:11.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2fbc156860 0x7f2fbc0c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fbc0ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fbc0ca270 con 0x7f2fbc15d080 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f2fbc0c97a0 con 0x7f2fbc156860 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc0ca530 con 0x7f2fbc0ce1c0 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbaa00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2fbc15d080 0x7f2fbc0c8e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51044/0 (socket says 172.21.15.81:51044) 2024-05-27T06:48:11.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbaa00640 1 -- 172.21.15.81:0/330914858 learned_addr learned my addr 172.21.15.81:0/330914858 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1866803024 0 0) 0x7f2fbc0ca270 con 0x7f2fbc15d080 2024-05-27T06:48:11.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fb3e00640 1 --2- 172.21.15.81:0/330914858 >> v2:172.21.15.106:3300/0 conn(0x7f2fbc156860 0x7f2fbc0c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbb400640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fbc0ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fa0003530 con 0x7f2fbc15d080 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 482258186 0 0) 0x7f2fa0003530 con 0x7f2fbc15d080 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 msgr2=0x7f2fbc0ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fbc0ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 >> v2:172.21.15.106:3300/0 conn(0x7f2fbc156860 msgr2=0x7f2fbc0c2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 --2- 172.21.15.81:0/330914858 >> v2:172.21.15.106:3300/0 conn(0x7f2fbc156860 0x7f2fbc0c2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc0caad0 con 0x7f2fbc15d080 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f2fac001700 con 0x7f2fbc15d080 2024-05-27T06:48:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 >> v1:172.21.15.81:6789/0 conn(0x7f2fbc15d080 legacy=0x7f2fbc0c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fa0003a10 0x7f2fa0003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fa0004810 0x7f2fa0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc0ca530 con 0x7f2fa0003a10 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa000bab0 con 0x7f2fa0004810 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa000bcb0 con 0x7f2fbc0ce1c0 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fa0003a10 0x7f2fa0003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fb3e00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fa0004810 0x7f2fa0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.154+0000 7f2fbb400640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 -- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 msgr2=0x7f2fa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fbc0ce1c0 0x7f2fa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 -- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fa0004810 msgr2=0x7f2fa0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:11.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fa0004810 0x7f2fa0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 -- 172.21.15.81:0/330914858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa0003530 con 0x7f2fa0003a10 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fa0003a10 0x7f2fa0003e80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c2c0 tx=0x7f2fac01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8c2c6dfb428753 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac007d60 con 0x7f2fa0003a10 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fac0a9070 con 0x7f2fa0003a10 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.158+0000 7f2fba000640 1 -- 172.21.15.81:0/330914858 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fac005d40 con 0x7f2fa0003a10 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- 172.21.15.81:0/330914858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fa0003a10 msgr2=0x7f2fa0003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/330914858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fa0003a10 0x7f2fa0003e80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c2c0 tx=0x7f2fac01cca0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbb400640 1 -- 172.21.15.81:0/330914858 reap_dead start 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- 172.21.15.81:0/330914858 shutdown_connections 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- 172.21.15.81:0/330914858 >> 172.21.15.81:0/330914858 conn(0x7f2fbc06c990 msgr2=0x7f2fbc06cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- 172.21.15.81:0/330914858 shutdown_connections 2024-05-27T06:48:11.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- 172.21.15.81:0/330914858 wait complete. 2024-05-27T06:48:11.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 Processor -- start 2024-05-27T06:48:11.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- start start 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fa0003a10 0x7f2fbc0c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fbc0c7310 0x7f2fbc14cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc15d490 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc0ce9e0 con 0x7f2fbc0c7310 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.162+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2fbc156c70 con 0x7f2fa0003a10 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42390/0 (socket says 172.21.15.81:42390) 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 -- 172.21.15.81:0/2978566328 learned_addr learned my addr 172.21.15.81:0/2978566328 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fbb400640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fbc0c7310 0x7f2fbc14cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fa0003a10 0x7f2fbc0c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 -- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fa0003a10 msgr2=0x7f2fbc0c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fa0003a10 0x7f2fbc0c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 -- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fbc0c7310 msgr2=0x7f2fbc14cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fbc0c7310 0x7f2fbc14cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 -- 172.21.15.81:0/2978566328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc0c6810 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb3e00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40119d0 tx=0x7f2fa4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400fb50 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fa400fcf0 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fbc0c5230 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400fea0 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fbc0c57f0 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fbc050870 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.166+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2fa4012040 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.170+0000 7f2fb2a00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 0x7f2f98071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:11.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.170+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 5939+0+0 (secure 0 0 0) 0x7f2fa408f830 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.170+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 0x7f2f98071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:11.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.170+0000 7f2fbaa00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 0x7f2f98071910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0ab000 tx=0x7f2fac0b1040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:11.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.174+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2fa4019030 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fbc0c9450 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fb2a00640 1 -- 172.21.15.81:0/2978566328 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6903 (secure 0 0 0) 0x7f2fa405c120 con 0x7f2fbc0c6f00 2024-05-27T06:48:11.361 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:11.362 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:47:49.316159+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"e","rank":4,"incarnation":18,"state":"up:active","state_seq":47,"addr":"172.21.15.81:6838/1291517441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":1291517441},{"type":"v1","addr":"172.21.15.81:6838","nonce":1291517441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4285,4281,4287,4235]},"id":1}]} 2024-05-27T06:48:11.362 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 23 2024-05-27T06:48:11.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 msgr2=0x7f2f98071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 0x7f2f98071910 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0ab000 tx=0x7f2fac0b1040 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 msgr2=0x7f2fbc14c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.366+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40119d0 tx=0x7f2fa4011e90 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 shutdown_connections 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2f9806f400 0x7f2f98071910 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fbc0c7310 0x7f2fbc14cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fbc0c6f00 0x7f2fbc14c7a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 --2- 172.21.15.81:0/2978566328 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2fa0003a10 0x7f2fbc0c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 >> 172.21.15.81:0/2978566328 conn(0x7f2fbc06c990 msgr2=0x7f2fbc073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 shutdown_connections 2024-05-27T06:48:11.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:11.370+0000 7f2fbbe00640 1 -- 172.21.15.81:0/2978566328 wait complete. 2024-05-27T06:48:11.383 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-05-27T06:48:13.329 INFO:tasks.workunit.client.0.smithi081.stdout:Create dir 200 to 299 ... 2024-05-27T06:48:13.331 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 200 2024-05-27T06:48:13.366 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 201 2024-05-27T06:48:13.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:13.398 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 202 2024-05-27T06:48:13.433 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 203 2024-05-27T06:48:13.466 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 204 2024-05-27T06:48:13.498 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 205 2024-05-27T06:48:13.528 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 206 2024-05-27T06:48:13.559 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 207 2024-05-27T06:48:13.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 Processor -- start 2024-05-27T06:48:13.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 -- start start 2024-05-27T06:48:13.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f26481569c0 0x7f2648073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26480ca810 0x7f26480cac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26480cb160 con 0x7f26480c9110 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f26480c89a0 con 0x7f26480ca810 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f264d200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f26480c8c80 con 0x7f26481569c0 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f2645400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26480ca810 0x7f26480cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f2645400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26480ca810 0x7f26480cac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42406/0 (socket says 172.21.15.81:42406) 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f2646800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f26480c9110 0x7f26480c9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51050/0 (socket says 172.21.15.81:51050) 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f2645400640 1 -- 172.21.15.81:0/3127493067 learned_addr learned my addr 172.21.15.81:0/3127493067 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.574+0000 7f2645e00640 1 --2- 172.21.15.81:0/3127493067 >> v2:172.21.15.106:3300/0 conn(0x7f26481569c0 0x7f2648073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2645400640 1 -- 172.21.15.81:0/3127493067 >> v2:172.21.15.106:3300/0 conn(0x7f26481569c0 msgr2=0x7f2648073d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:13.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2645400640 1 --2- 172.21.15.81:0/3127493067 >> v2:172.21.15.106:3300/0 conn(0x7f26481569c0 0x7f2648073d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2645400640 1 -- 172.21.15.81:0/3127493067 >> v1:172.21.15.81:6789/0 conn(0x7f26480c9110 legacy=0x7f26480c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2644a00640 1 -- 172.21.15.81:0/3127493067 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309371990 0 0) 0x7f26400014a0 con 0x7f26480c9110 2024-05-27T06:48:13.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2645400640 1 -- 172.21.15.81:0/3127493067 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26481606f0 con 0x7f26480ca810 2024-05-27T06:48:13.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2645400640 1 --2- 172.21.15.81:0/3127493067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f26480cac20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f26480cb640 tx=0x7f263401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de37e39855a24803 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:13.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2644a00640 1 -- 172.21.15.81:0/3127493067 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340b0070 con 0x7f26480ca810 2024-05-27T06:48:13.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2644a00640 1 -- 172.21.15.81:0/3127493067 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26340ab600 con 0x7f26480ca810 2024-05-27T06:48:13.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.578+0000 7f2644a00640 1 -- 172.21.15.81:0/3127493067 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340ab9d0 con 0x7f26480ca810 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- 172.21.15.81:0/3127493067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 msgr2=0x7f26480cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 --2- 172.21.15.81:0/3127493067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f26480cac20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f26480cb640 tx=0x7f263401c490 comp rx=0 tx=0).stop 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- 172.21.15.81:0/3127493067 shutdown_connections 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 --2- 172.21.15.81:0/3127493067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f26480cac20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 --2- 172.21.15.81:0/3127493067 >> v2:172.21.15.106:3300/0 conn(0x7f26481569c0 0x7f2648073d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- 172.21.15.81:0/3127493067 >> 172.21.15.81:0/3127493067 conn(0x7f2648075060 msgr2=0x7f2648070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:13.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- 172.21.15.81:0/3127493067 shutdown_connections 2024-05-27T06:48:13.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- 172.21.15.81:0/3127493067 wait complete. 2024-05-27T06:48:13.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 Processor -- start 2024-05-27T06:48:13.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.582+0000 7f264d200640 1 -- start start 2024-05-27T06:48:13.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26480c9110 0x7f26481490c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26481569c0 0x7f26481554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2648160c30 con 0x7f26480ca810 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2648160990 con 0x7f26481569c0 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2648068bd0 con 0x7f26480c9110 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42410/0 (socket says 172.21.15.81:42410) 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 -- 172.21.15.81:0/615150293 learned_addr learned my addr 172.21.15.81:0/615150293 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645e00640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26480c9110 0x7f26481490c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2646800640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26481569c0 0x7f26481554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 -- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26480c9110 msgr2=0x7f26481490c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26480c9110 0x7f26481490c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 -- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26481569c0 msgr2=0x7f26481554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26481569c0 0x7f26481554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 -- 172.21.15.81:0/615150293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26481f92d0 con 0x7f26480ca810 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f2645400640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f26340ae040 tx=0x7f2634019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340b0040 con 0x7f26480ca810 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26340b9070 con 0x7f26480ca810 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340b4680 con 0x7f26480ca810 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f264815f800 con 0x7f26480ca810 2024-05-27T06:48:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2648148070 con 0x7f26480ca810 2024-05-27T06:48:13.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.586+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264806a560 con 0x7f26480ca810 2024-05-27T06:48:13.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.590+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f26340b4820 con 0x7f26480ca810 2024-05-27T06:48:13.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.590+0000 7f263e000640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 0x7f2628071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:13.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.590+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f2634128ec0 con 0x7f26480ca810 2024-05-27T06:48:13.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.590+0000 7f2645e00640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 0x7f2628071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:13.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.590+0000 7f2645e00640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 0x7f2628071940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2638009ac0 tx=0x7f2638009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:13.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.594+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2634016110 con 0x7f26480ca810 2024-05-27T06:48:13.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.790+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f264815a110 con 0x7f26480ca810 2024-05-27T06:48:13.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.790+0000 7f263e000640 1 -- 172.21.15.81:0/615150293 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6073 (secure 0 0 0) 0x7f26340f5640 con 0x7f26480ca810 2024-05-27T06:48:13.787 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:13.787 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:13.298886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:48:13.788 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 24 2024-05-27T06:48:13.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 msgr2=0x7f2628071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 0x7f2628071940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2638009ac0 tx=0x7f2638009190 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 msgr2=0x7f264815f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f26340ae040 tx=0x7f2634019230 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 shutdown_connections 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f262806f430 0x7f2628071940 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26481569c0 0x7f26481554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26480ca810 0x7f264815f2c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 --2- 172.21.15.81:0/615150293 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26480c9110 0x7f26481490c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 >> 172.21.15.81:0/615150293 conn(0x7f2648075060 msgr2=0x7f2648075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 shutdown_connections 2024-05-27T06:48:13.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:13.794+0000 7f264d200640 1 -- 172.21.15.81:0/615150293 wait complete. 2024-05-27T06:48:13.810 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-05-27T06:48:13.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:48:13.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:13.942 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 208 2024-05-27T06:48:13.972 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 209 2024-05-27T06:48:14.003 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 210 2024-05-27T06:48:14.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 Processor -- start 2024-05-27T06:48:14.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 -- start start 2024-05-27T06:48:14.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 0x7f468c1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c1674a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f468c160c20 con 0x7f468c0caca0 2024-05-27T06:48:14.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f468c06eb20 con 0x7f468c0ca2f0 2024-05-27T06:48:14.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.018+0000 7f4693600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f468c06f910 con 0x7f468c165080 2024-05-27T06:48:14.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f468be00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f468c0caca0 0x7f468c0cfd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51058/0 (socket says 172.21.15.81:51058) 2024-05-27T06:48:14.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f468be00640 1 -- 172.21.15.81:0/3610213543 learned_addr learned my addr 172.21.15.81:0/3610213543 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:14.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4690c00640 1 --2- 172.21.15.81:0/3610213543 >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 0x7f468c1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4691600640 1 --2- 172.21.15.81:0/3610213543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c1674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f468b400640 1 -- 172.21.15.81:0/3610213543 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779746882 0 0) 0x7f468c160c20 con 0x7f468c0caca0 2024-05-27T06:48:14.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f468b400640 1 -- 172.21.15.81:0/3610213543 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4674003530 con 0x7f468c0caca0 2024-05-27T06:48:14.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4691600640 1 -- 172.21.15.81:0/3610213543 >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 msgr2=0x7f468c1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:14.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4691600640 1 --2- 172.21.15.81:0/3610213543 >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 0x7f468c1606e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4691600640 1 -- 172.21.15.81:0/3610213543 >> v1:172.21.15.81:6789/0 conn(0x7f468c0caca0 legacy=0x7f468c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4691600640 1 -- 172.21.15.81:0/3610213543 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c15c710 con 0x7f468c165080 2024-05-27T06:48:14.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.022+0000 7f4690c00640 1 --2- 172.21.15.81:0/3610213543 >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 0x7f468c1606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:14.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4691600640 1 --2- 172.21.15.81:0/3610213543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c1674a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4684009d90 tx=0x7f468401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9b6c4a0b5770bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f468b400640 1 -- 172.21.15.81:0/3610213543 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840b2070 con 0x7f468c165080 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f468b400640 1 -- 172.21.15.81:0/3610213543 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46840a5a30 con 0x7f468c165080 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f468b400640 1 -- 172.21.15.81:0/3610213543 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840a5e00 con 0x7f468c165080 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 -- 172.21.15.81:0/3610213543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 msgr2=0x7f468c1674a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 --2- 172.21.15.81:0/3610213543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c1674a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4684009d90 tx=0x7f468401c970 comp rx=0 tx=0).stop 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 -- 172.21.15.81:0/3610213543 shutdown_connections 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 --2- 172.21.15.81:0/3610213543 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c1674a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 --2- 172.21.15.81:0/3610213543 >> v2:172.21.15.106:3300/0 conn(0x7f468c0ca2f0 0x7f468c1606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 -- 172.21.15.81:0/3610213543 >> 172.21.15.81:0/3610213543 conn(0x7f468c068ae0 msgr2=0x7f468c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 -- 172.21.15.81:0/3610213543 shutdown_connections 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.026+0000 7f4693600640 1 -- 172.21.15.81:0/3610213543 wait complete. 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 Processor -- start 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- start start 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f468c0caca0 0x7f468c15b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f468c15cf90 con 0x7f468c0caca0 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f468c15ccb0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f468c06e220 con 0x7f468c165080 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f468be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f468c0caca0 0x7f468c15b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:54604/0 (socket says 172.21.15.81:54604) 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 -- 172.21.15.81:0/2184651461 learned_addr learned my addr 172.21.15.81:0/2184651461 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4691600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 -- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 msgr2=0x7f468c151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 -- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f468c0caca0 msgr2=0x7f468c15b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f468c0caca0 0x7f468c15b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 -- 172.21.15.81:0/2184651461 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c15e180 con 0x7f468c0ca2f0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4691600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4690c00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f468000e980 tx=0x7f468000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4680017070 con 0x7f468c0ca2f0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f468000ca00 con 0x7f468c0ca2f0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f468000cdb0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c159870 con 0x7f468c0ca2f0 2024-05-27T06:48:14.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.030+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c158cf0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.034+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c06a560 con 0x7f468c0ca2f0 2024-05-27T06:48:14.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.034+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f468001a070 con 0x7f468c0ca2f0 2024-05-27T06:48:14.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.034+0000 7f4688c00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 0x7f466c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:14.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.034+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f468008f250 con 0x7f468c0ca2f0 2024-05-27T06:48:14.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.034+0000 7f468be00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 0x7f466c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:14.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.038+0000 7f468be00640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 0x7f466c071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4678007a70 tx=0x7f46780079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:14.035 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 211 2024-05-27T06:48:14.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.042+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f468005b9c0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f468c15c4c0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4688c00640 1 -- 172.21.15.81:0/2184651461 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6073 (secure 0 0 0) 0x7f468005fcc0 con 0x7f468c0ca2f0 2024-05-27T06:48:14.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:14.230 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:13.298886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:48:14.230 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 24 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 msgr2=0x7f466c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 0x7f466c071940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4678007a70 tx=0x7f46780079c0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 msgr2=0x7f468c15dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f468000e980 tx=0x7f468000ee40 comp rx=0 tx=0).stop 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 shutdown_connections 2024-05-27T06:48:14.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f468c165080 0x7f468c151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f466c06f430 0x7f466c071940 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f468c0caca0 0x7f468c15b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.234+0000 7f4693600640 1 --2- 172.21.15.81:0/2184651461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f468c0ca2f0 0x7f468c15dc40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.238+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 >> 172.21.15.81:0/2184651461 conn(0x7f468c068ae0 msgr2=0x7f468c0745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.238+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 shutdown_connections 2024-05-27T06:48:14.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:14.238+0000 7f4693600640 1 -- 172.21.15.81:0/2184651461 wait complete. 2024-05-27T06:48:14.251 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4281": { "addr": "172.21.15.81:6835/2990302599", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2990302599, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2990302599, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:48:13.298886+0000", "qdb_cluster": [ 4281, 4285, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 4281, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:48:15.381 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 212 2024-05-27T06:48:15.410 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 213 2024-05-27T06:48:15.439 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 214 2024-05-27T06:48:15.468 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 215 2024-05-27T06:48:15.495 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 216 2024-05-27T06:48:15.523 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 217 2024-05-27T06:48:15.550 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 218 2024-05-27T06:48:15.578 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 219 2024-05-27T06:48:15.608 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 220 2024-05-27T06:48:16.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:16.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.406+0000 7f51a6000640 1 Processor -- start 2024-05-27T06:48:16.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.406+0000 7f51a6000640 1 -- start start 2024-05-27T06:48:16.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f51a6000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f51a01546f0 0x7f51a0072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:16.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f51a6000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a015db00 con 0x7f51a00732f0 2024-05-27T06:48:16.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f51a6000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a0073860 con 0x7f51a015a510 2024-05-27T06:48:16.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f51a6000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f51a015e6f0 con 0x7f51a01546f0 2024-05-27T06:48:16.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f5196a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f51a00732f0 0x7f51a00c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60814/0 (socket says 172.21.15.81:60814) 2024-05-27T06:48:16.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f5196a00640 1 -- 172.21.15.81:0/1318415059 learned_addr learned my addr 172.21.15.81:0/1318415059 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519f400640 1 --2- 172.21.15.81:0/1318415059 >> v2:172.21.15.106:3300/0 conn(0x7f51a01546f0 0x7f51a0072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284368602 0 0) 0x7f51a015db00 con 0x7f51a00732f0 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5184003530 con 0x7f51a00732f0 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378510654 0 0) 0x7f51a0073860 con 0x7f51a015a510 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a015db00 con 0x7f51a015a510 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519f400640 1 -- 172.21.15.81:0/1318415059 >> v1:172.21.15.106:6789/0 conn(0x7f51a015a510 legacy=0x7f51a015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519f400640 1 -- 172.21.15.81:0/1318415059 >> v1:172.21.15.81:6789/0 conn(0x7f51a00732f0 legacy=0x7f51a00c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.410+0000 7f519f400640 1 -- 172.21.15.81:0/1318415059 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a015ddc0 con 0x7f51a01546f0 2024-05-27T06:48:16.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f519f400640 1 --2- 172.21.15.81:0/1318415059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a0072db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6e21073ec035af server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:16.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900b4070 con 0x7f51a01546f0 2024-05-27T06:48:16.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51900af600 con 0x7f51a01546f0 2024-05-27T06:48:16.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f519ea00640 1 -- 172.21.15.81:0/1318415059 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900af9d0 con 0x7f51a01546f0 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 -- 172.21.15.81:0/1318415059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 msgr2=0x7f51a0072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 --2- 172.21.15.81:0/1318415059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a0072db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001f150 comp rx=0 tx=0).stop 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 -- 172.21.15.81:0/1318415059 shutdown_connections 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 --2- 172.21.15.81:0/1318415059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a0072db0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 -- 172.21.15.81:0/1318415059 >> 172.21.15.81:0/1318415059 conn(0x7f51a0070700 msgr2=0x7f51a00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 -- 172.21.15.81:0/1318415059 shutdown_connections 2024-05-27T06:48:16.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.414+0000 7f51a6000640 1 -- 172.21.15.81:0/1318415059 wait complete. 2024-05-27T06:48:16.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 Processor -- start 2024-05-27T06:48:16.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- start start 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a01f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a015a510 0x7f51a0148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51a015e380 con 0x7f51a00732f0 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f51a00755b0 con 0x7f51a01546f0 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f51a0154b00 con 0x7f51a015a510 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:16.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59074/0 (socket says 172.21.15.81:59074) 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 -- 172.21.15.81:0/3622533097 learned_addr learned my addr 172.21.15.81:0/3622533097 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f5196a00640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a01f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519fe00640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a015a510 0x7f51a0148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 -- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a015a510 msgr2=0x7f51a0148370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a015a510 0x7f51a0148370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 -- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 msgr2=0x7f51a01f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a01f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 -- 172.21.15.81:0/3622533097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a01f1f70 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f519f400640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51900b4040 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51900afe30 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5190009450 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a0152960 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.418+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a0152f20 con 0x7f51a00732f0 2024-05-27T06:48:16.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a006a560 con 0x7f51a00732f0 2024-05-27T06:48:16.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51900c5060 con 0x7f51a00732f0 2024-05-27T06:48:16.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f5197e00640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 0x7f517c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:16.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f519012cbe0 con 0x7f51a00732f0 2024-05-27T06:48:16.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f5196a00640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 0x7f517c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:16.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.422+0000 7f5196a00640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 0x7f517c071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5188007fd0 tx=0x7f5188009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:16.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.426+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5190018cb0 con 0x7f51a00732f0 2024-05-27T06:48:16.428 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 221 2024-05-27T06:48:16.456 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 222 2024-05-27T06:48:16.487 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 223 2024-05-27T06:48:16.519 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 224 2024-05-27T06:48:16.549 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 225 2024-05-27T06:48:16.580 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 226 2024-05-27T06:48:16.609 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 227 2024-05-27T06:48:16.639 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 228 2024-05-27T06:48:16.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.662+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51a0148bd0 con 0x7f51a00732f0 2024-05-27T06:48:16.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.662+0000 7f5197e00640 1 -- 172.21.15.81:0/3622533097 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6073 (secure 0 0 0) 0x7f51900f92c0 con 0x7f51a00732f0 2024-05-27T06:48:16.659 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:16.659 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:13.298886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:48:16.659 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 24 2024-05-27T06:48:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 msgr2=0x7f517c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 0x7f517c071910 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5188007fd0 tx=0x7f5188009040 comp rx=0 tx=0).stop 2024-05-27T06:48:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 msgr2=0x7f51a0153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5190009d90 tx=0x7f519001bef0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 shutdown_connections 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51a015a510 0x7f51a0148370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f517c06f400 0x7f517c071910 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51a01546f0 0x7f51a01f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 --2- 172.21.15.81:0/3622533097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51a00732f0 0x7f51a0153a00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 >> 172.21.15.81:0/3622533097 conn(0x7f51a0070700 msgr2=0x7f51a0159770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 shutdown_connections 2024-05-27T06:48:16.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:16.666+0000 7f51a6000640 1 -- 172.21.15.81:0/3622533097 wait complete. 2024-05-27T06:48:16.670 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 229 2024-05-27T06:48:16.687 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:16.704 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 230 2024-05-27T06:48:16.738 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 231 2024-05-27T06:48:16.771 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 232 2024-05-27T06:48:16.802 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 233 2024-05-27T06:48:16.833 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 234 2024-05-27T06:48:16.864 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 235 2024-05-27T06:48:16.893 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 236 2024-05-27T06:48:17.524 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 237 2024-05-27T06:48:17.557 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 238 2024-05-27T06:48:18.113 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 239 2024-05-27T06:48:18.145 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 240 2024-05-27T06:48:18.178 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 241 2024-05-27T06:48:18.209 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 242 2024-05-27T06:48:18.538 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 243 2024-05-27T06:48:18.566 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 244 2024-05-27T06:48:18.595 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 245 2024-05-27T06:48:18.625 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 246 2024-05-27T06:48:18.653 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 247 2024-05-27T06:48:18.682 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 248 2024-05-27T06:48:18.689 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:18.711 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 249 2024-05-27T06:48:18.780 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 250 2024-05-27T06:48:18.812 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 251 2024-05-27T06:48:18.844 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 252 2024-05-27T06:48:18.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 Processor -- start 2024-05-27T06:48:18.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 -- start start 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ab80ca6b0 0x7f1ab8070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab8154c10 con 0x7f1ab80cbe90 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab815b050 con 0x7f1ab8070ac0 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1abce00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab80caac0 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ab80ca6b0 0x7f1ab8070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ab80ca6b0 0x7f1ab8070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59090/0 (socket says 172.21.15.81:59090) 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1ab80cbe90 0x7f1ab8072100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60828/0 (socket says 172.21.15.81:60828) 2024-05-27T06:48:18.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 -- 172.21.15.81:0/3110857661 learned_addr learned my addr 172.21.15.81:0/3110857661 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:18.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 -- 172.21.15.81:0/3110857661 >> v1:172.21.15.106:6789/0 conn(0x7f1ab8070ac0 legacy=0x7f1ab8164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:18.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 -- 172.21.15.81:0/3110857661 >> v1:172.21.15.81:6789/0 conn(0x7f1ab80cbe90 legacy=0x7f1ab8072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:18.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.870+0000 7f1ab5400640 1 -- 172.21.15.81:0/3110857661 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab815b330 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.874+0000 7f1ab5400640 1 --2- 172.21.15.81:0/3110857661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab8070580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009870 tx=0x7f1aa401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9594a258cc0cddb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:18.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.874+0000 7f1ab4a00640 1 -- 172.21.15.81:0/3110857661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40ab620 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.874+0000 7f1ab4a00640 1 -- 172.21.15.81:0/3110857661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa40b0070 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.874+0000 7f1ab4a00640 1 -- 172.21.15.81:0/3110857661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40ab9e0 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- 172.21.15.81:0/3110857661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 msgr2=0x7f1ab8070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:18.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 --2- 172.21.15.81:0/3110857661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab8070580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009870 tx=0x7f1aa401c480 comp rx=0 tx=0).stop 2024-05-27T06:48:18.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- 172.21.15.81:0/3110857661 shutdown_connections 2024-05-27T06:48:18.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 --2- 172.21.15.81:0/3110857661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab8070580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:18.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- 172.21.15.81:0/3110857661 >> 172.21.15.81:0/3110857661 conn(0x7f1ab8074c80 msgr2=0x7f1ab80cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:18.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- 172.21.15.81:0/3110857661 shutdown_connections 2024-05-27T06:48:18.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- 172.21.15.81:0/3110857661 wait complete. 2024-05-27T06:48:18.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 Processor -- start 2024-05-27T06:48:18.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.878+0000 7f1abce00640 1 -- start start 2024-05-27T06:48:18.875 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 253 2024-05-27T06:48:18.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab8070ac0 0x7f1ab81f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:18.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab81f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:18.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:18.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80c9d00 con 0x7f1ab80ca6b0 2024-05-27T06:48:18.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80c9a20 con 0x7f1ab80cbe90 2024-05-27T06:48:18.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab8070ed0 con 0x7f1ab8070ac0 2024-05-27T06:48:18.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:18.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab8070ac0 0x7f1ab81f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46798/0 (socket says 172.21.15.81:46798) 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 -- 172.21.15.81:0/3718543617 learned_addr learned my addr 172.21.15.81:0/3718543617 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab5400640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab81f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 -- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab8070ac0 msgr2=0x7f1ab81f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab8070ac0 0x7f1ab81f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 -- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 msgr2=0x7f1ab81f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab81f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 -- 172.21.15.81:0/3718543617 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab815cab0 con 0x7f1ab80cbe90 2024-05-27T06:48:18.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1ab6800640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c6b0 tx=0x7f1ab000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:18.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003b80 con 0x7f1ab80cbe90 2024-05-27T06:48:18.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab0003d20 con 0x7f1ab80cbe90 2024-05-27T06:48:18.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab81f2520 con 0x7f1ab80cbe90 2024-05-27T06:48:18.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00052f0 con 0x7f1ab80cbe90 2024-05-27T06:48:18.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.882+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab815bc40 con 0x7f1ab80cbe90 2024-05-27T06:48:18.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab815b9b0 con 0x7f1ab80cbe90 2024-05-27T06:48:18.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1ab0005490 con 0x7f1ab80cbe90 2024-05-27T06:48:18.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1aae000640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 0x7f1a98071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:18.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f1ab008f0d0 con 0x7f1ab80cbe90 2024-05-27T06:48:18.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1ab5e00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 0x7f1a98071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:18.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.886+0000 7f1ab5e00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 0x7f1a98071940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8000f70 tx=0x7f1aa800ae30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:18.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:18.894+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1ab005b840 con 0x7f1ab80cbe90 2024-05-27T06:48:18.904 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 254 2024-05-27T06:48:18.931 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 255 2024-05-27T06:48:18.961 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 256 2024-05-27T06:48:18.990 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 257 2024-05-27T06:48:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.110+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ab806a560 con 0x7f1ab80cbe90 2024-05-27T06:48:19.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.110+0000 7f1aae000640 1 -- 172.21.15.81:0/3718543617 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6074 (secure 0 0 0) 0x7f1ab005fb40 con 0x7f1ab80cbe90 2024-05-27T06:48:19.106 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:19.107 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:18.312972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:resolve","state_seq":8,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:48:19.107 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 msgr2=0x7f1a98071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 0x7f1a98071940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8000f70 tx=0x7f1aa800ae30 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 msgr2=0x7f1ab814c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c6b0 tx=0x7f1ab000cb70 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 shutdown_connections 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1a9806f430 0x7f1a98071940 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ab80cbe90 0x7f1ab814c0e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab80ca6b0 0x7f1ab81f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 --2- 172.21.15.81:0/3718543617 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ab8070ac0 0x7f1ab81f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 >> 172.21.15.81:0/3718543617 conn(0x7f1ab8074c80 msgr2=0x7f1ab814e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 shutdown_connections 2024-05-27T06:48:19.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:19.114+0000 7f1abce00640 1 -- 172.21.15.81:0/3718543617 wait complete. 2024-05-27T06:48:19.128 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:19.185 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 258 2024-05-27T06:48:19.213 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 259 2024-05-27T06:48:19.598 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 260 2024-05-27T06:48:20.178 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 261 2024-05-27T06:48:20.207 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 262 2024-05-27T06:48:20.236 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 263 2024-05-27T06:48:20.266 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 264 2024-05-27T06:48:20.295 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 265 2024-05-27T06:48:20.325 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 266 2024-05-27T06:48:20.356 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 267 2024-05-27T06:48:20.386 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 268 2024-05-27T06:48:20.415 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 269 2024-05-27T06:48:20.443 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 270 2024-05-27T06:48:20.471 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 271 2024-05-27T06:48:20.499 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 272 2024-05-27T06:48:20.723 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 273 2024-05-27T06:48:21.105 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 274 2024-05-27T06:48:21.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:21.133 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 275 2024-05-27T06:48:21.162 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 276 2024-05-27T06:48:21.191 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 277 2024-05-27T06:48:21.218 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 278 2024-05-27T06:48:21.247 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 279 2024-05-27T06:48:21.274 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 280 2024-05-27T06:48:21.302 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 281 2024-05-27T06:48:21.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 Processor -- start 2024-05-27T06:48:21.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 -- start start 2024-05-27T06:48:21.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa3a80ca770 0x7fa3a80cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a80c78b0 con 0x7fa3a80d02f0 2024-05-27T06:48:21.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a80c6ee0 con 0x7fa3a80cbd00 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3ad800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a80c71c0 con 0x7fa3a80ca770 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59108/0 (socket says 172.21.15.81:59108) 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 -- 172.21.15.81:0/2774947702 learned_addr learned my addr 172.21.15.81:0/2774947702 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3a6a00640 1 --2- 172.21.15.81:0/2774947702 >> v2:172.21.15.106:3300/0 conn(0x7fa3a80ca770 0x7fa3a80cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 -- 172.21.15.81:0/2774947702 >> v2:172.21.15.106:3300/0 conn(0x7fa3a80ca770 msgr2=0x7fa3a80cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 --2- 172.21.15.81:0/2774947702 >> v2:172.21.15.106:3300/0 conn(0x7fa3a80ca770 0x7fa3a80cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 -- 172.21.15.81:0/2774947702 >> v1:172.21.15.106:6789/0 conn(0x7fa3a80d02f0 legacy=0x7fa3a8154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa39e000640 1 -- 172.21.15.81:0/2774947702 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a80c7b70 con 0x7fa3a80cbd00 2024-05-27T06:48:21.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.314+0000 7fa3a6000640 1 -- 172.21.15.81:0/2774947702 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887003428 0 0) 0x7fa3a80c78b0 con 0x7fa3a80d02f0 2024-05-27T06:48:21.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.318+0000 7fa39e000640 1 --2- 172.21.15.81:0/2774947702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8073da0 tx=0x7fa39001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9977b1f29cab3235 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:21.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.318+0000 7fa3a6000640 1 -- 172.21.15.81:0/2774947702 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3900ab430 con 0x7fa3a80cbd00 2024-05-27T06:48:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.318+0000 7fa3a6000640 1 -- 172.21.15.81:0/2774947702 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3900b0070 con 0x7fa3a80cbd00 2024-05-27T06:48:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.318+0000 7fa3a6000640 1 -- 172.21.15.81:0/2774947702 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3900ab7f0 con 0x7fa3a80cbd00 2024-05-27T06:48:21.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- 172.21.15.81:0/2774947702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80cbd00 msgr2=0x7fa3a80cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2774947702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8073da0 tx=0x7fa39001c520 comp rx=0 tx=0).stop 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- 172.21.15.81:0/2774947702 shutdown_connections 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2774947702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80cbd00 0x7fa3a80cfdb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2774947702 >> v2:172.21.15.106:3300/0 conn(0x7fa3a80ca770 0x7fa3a80cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- 172.21.15.81:0/2774947702 >> 172.21.15.81:0/2774947702 conn(0x7fa3a8068ae0 msgr2=0x7fa3a8068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- 172.21.15.81:0/2774947702 shutdown_connections 2024-05-27T06:48:21.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- 172.21.15.81:0/2774947702 wait complete. 2024-05-27T06:48:21.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 Processor -- start 2024-05-27T06:48:21.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.322+0000 7fa3ad800640 1 -- start start 2024-05-27T06:48:21.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a80cbd00 0x7fa3a8155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a80d02f0 0x7fa3a806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8167590 con 0x7fa3a80ca770 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8154b20 con 0x7fa3a80d02f0 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a8167710 con 0x7fa3a80cbd00 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa39e000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a80cbd00 0x7fa3a8155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59112/0 (socket says 172.21.15.81:59112) 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 -- 172.21.15.81:0/2003315361 learned_addr learned my addr 172.21.15.81:0/2003315361 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a7400640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a80d02f0 0x7fa3a806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 -- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a80cbd00 msgr2=0x7fa3a8155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a80cbd00 0x7fa3a8155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 -- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a80d02f0 msgr2=0x7fa3a806f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a80d02f0 0x7fa3a806f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 -- 172.21.15.81:0/2003315361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a806d8a0 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3a6a00640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa398009d10 tx=0x7fa39800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398018020 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa398014480 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a814f150 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398014830 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a80cba70 con 0x7fa3a80ca770 2024-05-27T06:48:21.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.326+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a806a560 con 0x7fa3a80ca770 2024-05-27T06:48:21.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.330+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa39801b020 con 0x7fa3a80ca770 2024-05-27T06:48:21.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.330+0000 7fa39f400640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 0x7fa384071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:21.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.330+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fa39808fe90 con 0x7fa3a80ca770 2024-05-27T06:48:21.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.330+0000 7fa39e000640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 0x7fa384071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:21.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.330+0000 7fa39e000640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 0x7fa384071990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3900af000 tx=0x7fa3900a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:21.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.338+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa398022030 con 0x7fa3a80ca770 2024-05-27T06:48:21.332 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 282 2024-05-27T06:48:21.361 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 283 2024-05-27T06:48:21.390 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 284 2024-05-27T06:48:21.420 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 285 2024-05-27T06:48:21.450 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 286 2024-05-27T06:48:21.477 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 287 2024-05-27T06:48:21.504 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 288 2024-05-27T06:48:21.533 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 289 2024-05-27T06:48:21.561 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 290 2024-05-27T06:48:21.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.578+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3a80c78b0 con 0x7fa3a80ca770 2024-05-27T06:48:21.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa39f400640 1 -- 172.21.15.81:0/2003315361 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6074 (secure 0 0 0) 0x7fa398010040 con 0x7fa3a80ca770 2024-05-27T06:48:21.575 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:21.576 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:18.312972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:resolve","state_seq":8,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:48:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-05-27T06:48:21.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 msgr2=0x7fa384071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 0x7fa384071990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3900af000 tx=0x7fa3900a7000 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 msgr2=0x7fa3a81f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa398009d10 tx=0x7fa39800c490 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 shutdown_connections 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa3a80cbd00 0x7fa3a8155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa38406f480 0x7fa384071990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3a80d02f0 0x7fa3a806f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 --2- 172.21.15.81:0/2003315361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3a80ca770 0x7fa3a81f6290 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 >> 172.21.15.81:0/2003315361 conn(0x7fa3a8068ae0 msgr2=0x7fa3a8152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 shutdown_connections 2024-05-27T06:48:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:21.582+0000 7fa3ad800640 1 -- 172.21.15.81:0/2003315361 wait complete. 2024-05-27T06:48:21.591 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 291 2024-05-27T06:48:21.601 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:21.622 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 292 2024-05-27T06:48:21.651 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 293 2024-05-27T06:48:21.682 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 294 2024-05-27T06:48:22.056 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 295 2024-05-27T06:48:23.052 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 296 2024-05-27T06:48:23.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:23.747 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 297 2024-05-27T06:48:23.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.766+0000 7f1af0a00640 1 Processor -- start 2024-05-27T06:48:23.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.766+0000 7f1af0a00640 1 -- start start 2024-05-27T06:48:23.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1af0a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1af0a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1af0a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aec15eb60 con 0x7f1aec071f20 2024-05-27T06:48:23.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1af0a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1aec0ca380 con 0x7f1aec154790 2024-05-27T06:48:23.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1af0a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec0ca660 con 0x7f1aec0caef0 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1ae1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aeb200640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f1aec071f20 0x7f1aec074530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:57336/0 (socket says 172.21.15.81:57336) 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aeb200640 1 -- 172.21.15.81:0/1789123631 learned_addr learned my addr 172.21.15.81:0/1789123631 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1ae1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46830/0 (socket says 172.21.15.81:46830) 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:57676/0 (socket says 172.21.15.81:57676) 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1ae9e00640 1 -- 172.21.15.81:0/1789123631 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116204928 0 0) 0x7f1aec15eb60 con 0x7f1aec071f20 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1ae9e00640 1 -- 172.21.15.81:0/1789123631 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ad0003530 con 0x7f1aec071f20 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 -- 172.21.15.81:0/1789123631 >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 msgr2=0x7f1aec0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 --2- 172.21.15.81:0/1789123631 >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 -- 172.21.15.81:0/1789123631 >> v1:172.21.15.106:6789/0 conn(0x7f1aec071f20 legacy=0x7f1aec074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1aea800640 1 -- 172.21.15.81:0/1789123631 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec0ca940 con 0x7f1aec0caef0 2024-05-27T06:48:23.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.770+0000 7f1ae1e00640 1 --2- 172.21.15.81:0/1789123631 >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1aea800640 1 --2- 172.21.15.81:0/1789123631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0097c0 tx=0x7f1adc01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7dbd31282b72ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1ae9e00640 1 -- 172.21.15.81:0/1789123631 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc0b5070 con 0x7f1aec0caef0 2024-05-27T06:48:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1ae9e00640 1 -- 172.21.15.81:0/1789123631 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1adc0a8990 con 0x7f1aec0caef0 2024-05-27T06:48:23.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1ae9e00640 1 -- 172.21.15.81:0/1789123631 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc0a8d40 con 0x7f1aec0caef0 2024-05-27T06:48:23.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 -- 172.21.15.81:0/1789123631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 msgr2=0x7f1aec15e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 --2- 172.21.15.81:0/1789123631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0097c0 tx=0x7f1adc01ec90 comp rx=0 tx=0).stop 2024-05-27T06:48:23.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 -- 172.21.15.81:0/1789123631 shutdown_connections 2024-05-27T06:48:23.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 --2- 172.21.15.81:0/1789123631 >> v2:172.21.15.106:3300/0 conn(0x7f1aec154790 0x7f1aec0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 --2- 172.21.15.81:0/1789123631 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec0caef0 0x7f1aec15e620 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 -- 172.21.15.81:0/1789123631 >> 172.21.15.81:0/1789123631 conn(0x7f1aec068d90 msgr2=0x7f1aec0cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 -- 172.21.15.81:0/1789123631 shutdown_connections 2024-05-27T06:48:23.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.774+0000 7f1af0a00640 1 -- 172.21.15.81:0/1789123631 wait complete. 2024-05-27T06:48:23.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 Processor -- start 2024-05-27T06:48:23.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- start start 2024-05-27T06:48:23.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 0x7f1aec075fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec154790 0x7f1aec149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec0cbfe0 con 0x7f1aec0caef0 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec15e100 con 0x7f1aec071f20 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec154ba0 con 0x7f1aec154790 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1aea800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 0x7f1aec075fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1aea800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46834/0 (socket says 172.21.15.81:46834) 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1aea800640 1 -- 172.21.15.81:0/2759925980 learned_addr learned my addr 172.21.15.81:0/2759925980 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1aeb200640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec154790 0x7f1aec149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 -- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec154790 msgr2=0x7f1aec149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec154790 0x7f1aec149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 -- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 msgr2=0x7f1aec14d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 -- 172.21.15.81:0/2759925980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec14d5a0 con 0x7f1aec0caef0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1aea800640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae1e00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 0x7f1aec075fb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1aec073fe0 tx=0x7f1ad400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4017070 con 0x7f1aec0caef0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad400c8e0 con 0x7f1aec0caef0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400ccd0 con 0x7f1aec0caef0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec15d9e0 con 0x7f1aec0caef0 2024-05-27T06:48:23.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.778+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec1f1e50 con 0x7f1aec0caef0 2024-05-27T06:48:23.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec050870 con 0x7f1aec0caef0 2024-05-27T06:48:23.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1ad401a070 con 0x7f1aec0caef0 2024-05-27T06:48:23.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1ae3400640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 0x7f1ac8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:23.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f1ad408f450 con 0x7f1aec0caef0 2024-05-27T06:48:23.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1aea800640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 0x7f1ac8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:23.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.782+0000 7f1aea800640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 0x7f1ac8071940 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0097c0 tx=0x7f1adc0aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:23.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.786+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1ad405bb30 con 0x7f1aec0caef0 2024-05-27T06:48:23.969 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 298 2024-05-27T06:48:23.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.986+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aec158040 con 0x7f1aec0caef0 2024-05-27T06:48:23.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.986+0000 7f1ae3400640 1 -- 172.21.15.81:0/2759925980 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6074 (secure 0 0 0) 0x7f1ad4064030 con 0x7f1aec0caef0 2024-05-27T06:48:23.982 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:23.982 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:18.312972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:resolve","state_seq":8,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:48:23.983 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 25 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 msgr2=0x7f1ac8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 0x7f1ac8071940 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0097c0 tx=0x7f1adc0aa070 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 msgr2=0x7f1aec075fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 0x7f1aec075fb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1aec073fe0 tx=0x7f1ad400ed40 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 shutdown_connections 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1aec154790 0x7f1aec149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ac806f430 0x7f1ac8071940 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aec0caef0 0x7f1aec075fb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 --2- 172.21.15.81:0/2759925980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1aec071f20 0x7f1aec14d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:23.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 >> 172.21.15.81:0/2759925980 conn(0x7f1aec068d90 msgr2=0x7f1aec15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 shutdown_connections 2024-05-27T06:48:23.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:23.990+0000 7f1af0a00640 1 -- 172.21.15.81:0/2759925980 wait complete. 2024-05-27T06:48:24.000 INFO:tasks.workunit.client.0.smithi081.stdout: create dir 299 2024-05-27T06:48:24.007 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:24.029 INFO:tasks.workunit.client.0.smithi081.stdout:Create a snapshot in every first level dir ... 2024-05-27T06:48:24.138 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/.snap/snap-subdir-test 2024-05-27T06:48:24.627 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/30 file after the snapshot 2024-05-27T06:48:24.629 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/31 file after the snapshot 2024-05-27T06:48:24.632 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/32 file after the snapshot 2024-05-27T06:48:24.634 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/33 file after the snapshot 2024-05-27T06:48:24.637 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/34 file after the snapshot 2024-05-27T06:48:24.639 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/35 file after the snapshot 2024-05-27T06:48:24.642 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/36 file after the snapshot 2024-05-27T06:48:24.644 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/37 file after the snapshot 2024-05-27T06:48:24.647 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/38 file after the snapshot 2024-05-27T06:48:24.649 INFO:tasks.workunit.client.0.smithi081.stdout: create 100/39 file after the snapshot 2024-05-27T06:48:24.652 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/.snap/snap-subdir-test 2024-05-27T06:48:26.008 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:26.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 Processor -- start 2024-05-27T06:48:26.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 -- start start 2024-05-27T06:48:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9c072840 con 0x7f7f9c15a040 2024-05-27T06:48:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7f9c0c5040 con 0x7f7f9c161cd0 2024-05-27T06:48:26.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa3e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9c0c75c0 con 0x7f7f9c072430 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa0a00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f7f9c15a040 0x7f7f9c0c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:36514/0 (socket says 172.21.15.81:36514) 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa0a00640 1 -- 172.21.15.81:0/3558350902 learned_addr learned my addr 172.21.15.81:0/3558350902 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:37498/0 (socket says 172.21.15.81:37498) 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:46816/0 (socket says 172.21.15.81:46816) 2024-05-27T06:48:26.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7f9be00640 1 -- 172.21.15.81:0/3558350902 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2049212039 0 0) 0x7f7f9c072840 con 0x7f7f9c15a040 2024-05-27T06:48:26.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 -- 172.21.15.81:0/3558350902 >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 msgr2=0x7f7f9c1640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:26.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 --2- 172.21.15.81:0/3558350902 >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 -- 172.21.15.81:0/3558350902 >> v1:172.21.15.106:6789/0 conn(0x7f7f9c15a040 legacy=0x7f7f9c0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1400640 1 -- 172.21.15.81:0/3558350902 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9c0c5270 con 0x7f7f9c072430 2024-05-27T06:48:26.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.174+0000 7f7fa1e00640 1 --2- 172.21.15.81:0/3558350902 >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7fa1400640 1 --2- 172.21.15.81:0/3558350902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f900097c0 tx=0x7f7f9001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f663374dff2142 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7f9be00640 1 -- 172.21.15.81:0/3558350902 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b0070 con 0x7f7f9c072430 2024-05-27T06:48:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7f9be00640 1 -- 172.21.15.81:0/3558350902 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f900ab600 con 0x7f7f9c072430 2024-05-27T06:48:26.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7f9be00640 1 -- 172.21.15.81:0/3558350902 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900ab9d0 con 0x7f7f9c072430 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7fa3e00640 1 -- 172.21.15.81:0/3558350902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 msgr2=0x7f7f9c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.178+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/3558350902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f900097c0 tx=0x7f7f9001c4c0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 -- 172.21.15.81:0/3558350902 shutdown_connections 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/3558350902 >> v2:172.21.15.106:3300/0 conn(0x7f7f9c161cd0 0x7f7f9c1640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/3558350902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c072430 0x7f7f9c159b00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 -- 172.21.15.81:0/3558350902 >> 172.21.15.81:0/3558350902 conn(0x7f7f9c070700 msgr2=0x7f7f9c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 -- 172.21.15.81:0/3558350902 shutdown_connections 2024-05-27T06:48:26.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 -- 172.21.15.81:0/3558350902 wait complete. 2024-05-27T06:48:26.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 Processor -- start 2024-05-27T06:48:26.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 -- start start 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.182+0000 7f7fa3e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9c15a040 0x7f7f9c14c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c161cd0 0x7f7f9c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c0ce6f0 con 0x7f7f9c072430 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c0c7dd0 con 0x7f7f9c15a040 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9c0c5610 con 0x7f7f9c161cd0 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42074/0 (socket says 172.21.15.81:42074) 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 -- 172.21.15.81:0/2743381292 learned_addr learned my addr 172.21.15.81:0/2743381292 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa0a00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9c15a040 0x7f7f9c14c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c161cd0 0x7f7f9c1584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 -- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c161cd0 msgr2=0x7f7f9c1584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c161cd0 0x7f7f9c1584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 -- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9c15a040 msgr2=0x7f7f9c14c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9c15a040 0x7f7f9c14c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 -- 172.21.15.81:0/2743381292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9c1f6830 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa1400640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f9001a0f0 tx=0x7f7f90019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b0040 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f900b9070 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9c14c650 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b4670 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9c1564d0 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9c06a560 con 0x7f7f9c072430 2024-05-27T06:48:26.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.186+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7f900b4810 con 0x7f7f9c072430 2024-05-27T06:48:26.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.190+0000 7f7f99600640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 0x7f7f7c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:26.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.190+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f7f90128f80 con 0x7f7f9c072430 2024-05-27T06:48:26.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.190+0000 7f7fa0a00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 0x7f7f7c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:26.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.190+0000 7f7fa0a00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 0x7f7f7c071940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c15da20 tx=0x7f7f88007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:26.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.194+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7f900160b0 con 0x7f7f9c072430 2024-05-27T06:48:26.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.382+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f9c050d90 con 0x7f7f9c072430 2024-05-27T06:48:26.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.382+0000 7f7f99600640 1 -- 172.21.15.81:0/2743381292 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6077 (secure 0 0 0) 0x7f7f900f5d40 con 0x7f7f9c072430 2024-05-27T06:48:26.380 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:26.380 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:23.732532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:48:26.381 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 26 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 msgr2=0x7f7f7c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 0x7f7f7c071940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c15da20 tx=0x7f7f88007df0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 msgr2=0x7f7f9c1f62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7f9001a0f0 tx=0x7f7f90019260 comp rx=0 tx=0).stop 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 shutdown_connections 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f9c161cd0 0x7f7f9c1584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f7c06f430 0x7f7f7c071940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f9c15a040 0x7f7f9c14c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 --2- 172.21.15.81:0/2743381292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f9c072430 0x7f7f9c1f62f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:26.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 >> 172.21.15.81:0/2743381292 conn(0x7f7f9c070700 msgr2=0x7f7f9c0c38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:26.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 shutdown_connections 2024-05-27T06:48:26.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:26.386+0000 7f7fa3e00640 1 -- 172.21.15.81:0/2743381292 wait complete. 2024-05-27T06:48:26.401 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4281": { "addr": "172.21.15.81:6835/2990302599", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2990302599, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2990302599, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4281, "incarnation": 14, "join_fscid": -1, "name": "c", "rank": 3, "state": "up:active", "state_seq": 9 }, "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:48:23.732532+0000", "qdb_cluster": [ 4281, 4285, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 4281, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:48:28.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:28.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 Processor -- start 2024-05-27T06:48:28.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 -- start start 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f7480072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f74800732f0 0x7f74800c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74800c7510 0x7f748015a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f748015e6f0 con 0x7f74800c7510 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f74800755b0 con 0x7f74800732f0 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f7485600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f748015db00 con 0x7f74801546f0 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f747f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74800c7510 0x7f748015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f747f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74800c7510 0x7f748015a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42082/0 (socket says 172.21.15.81:42082) 2024-05-27T06:48:28.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.566+0000 7f747f400640 1 -- 172.21.15.81:0/2471289203 learned_addr learned my addr 172.21.15.81:0/2471289203 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f7476000640 1 --2- 172.21.15.81:0/2471289203 >> v2:172.21.15.106:3300/0 conn(0x7f74800732f0 0x7f74800c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747ea00640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f7480072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 -- 172.21.15.81:0/2471289203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 msgr2=0x7f7480072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f7480072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 -- 172.21.15.81:0/2471289203 >> v2:172.21.15.106:3300/0 conn(0x7f74800732f0 msgr2=0x7f74800c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 --2- 172.21.15.81:0/2471289203 >> v2:172.21.15.106:3300/0 conn(0x7f74800732f0 0x7f74800c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 -- 172.21.15.81:0/2471289203 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7480073860 con 0x7f74800c7510 2024-05-27T06:48:28.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747f400640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800c7510 0x7f748015a510 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f748006d860 tx=0x7f747801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7ceefb6a6a95bb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:28.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747e000640 1 -- 172.21.15.81:0/2471289203 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74780b5070 con 0x7f74800c7510 2024-05-27T06:48:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747e000640 1 -- 172.21.15.81:0/2471289203 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74780a89d0 con 0x7f74800c7510 2024-05-27T06:48:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.570+0000 7f747e000640 1 -- 172.21.15.81:0/2471289203 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74780a8da0 con 0x7f74800c7510 2024-05-27T06:48:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- 172.21.15.81:0/2471289203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800c7510 msgr2=0x7f748015a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800c7510 0x7f748015a510 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f748006d860 tx=0x7f747801ed10 comp rx=0 tx=0).stop 2024-05-27T06:48:28.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- 172.21.15.81:0/2471289203 shutdown_connections 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800c7510 0x7f748015a510 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- 172.21.15.81:0/2471289203 >> v2:172.21.15.106:3300/0 conn(0x7f74800732f0 0x7f74800c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- 172.21.15.81:0/2471289203 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f7480072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- 172.21.15.81:0/2471289203 >> 172.21.15.81:0/2471289203 conn(0x7f7480070700 msgr2=0x7f74800cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- 172.21.15.81:0/2471289203 shutdown_connections 2024-05-27T06:48:28.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- 172.21.15.81:0/2471289203 wait complete. 2024-05-27T06:48:28.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 Processor -- start 2024-05-27T06:48:28.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 -- start start 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.574+0000 7f7485600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74800c7510 0x7f74800cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f74800c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f748015e380 con 0x7f74800732f0 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f748015e0a0 con 0x7f74800c7510 2024-05-27T06:48:28.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7480154b00 con 0x7f74801546f0 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42088/0 (socket says 172.21.15.81:42088) 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 -- 172.21.15.81:0/4206300340 learned_addr learned my addr 172.21.15.81:0/4206300340 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747f400640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f74800c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7476000640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74800c7510 0x7f74800cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 -- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 msgr2=0x7f74800c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f74800c3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 -- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74800c7510 msgr2=0x7f74800cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:28.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74800c7510 0x7f74800cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 -- 172.21.15.81:0/4206300340 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74800bf9a0 con 0x7f74800732f0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f747ea00640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7470007e70 tx=0x7f747000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470014620 con 0x7f74800732f0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7470019070 con 0x7f74800732f0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470014a00 con 0x7f74800732f0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74801f2f60 con 0x7f74800732f0 2024-05-27T06:48:28.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74801f3500 con 0x7f74800732f0 2024-05-27T06:48:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f748006a560 con 0x7f74800732f0 2024-05-27T06:48:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.578+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7470014ba0 con 0x7f74800732f0 2024-05-27T06:48:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.582+0000 7f7477400640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 0x7f745c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:28.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.582+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f747008fbe0 con 0x7f74800732f0 2024-05-27T06:48:28.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.582+0000 7f7476000640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 0x7f745c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:28.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.582+0000 7f7476000640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 0x7f745c071940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f748015d3d0 tx=0x7f7468008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:28.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.586+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f747005c9a0 con 0x7f74800732f0 2024-05-27T06:48:28.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74800c8e20 con 0x7f74800732f0 2024-05-27T06:48:28.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7477400640 1 -- 172.21.15.81:0/4206300340 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6077 (secure 0 0 0) 0x7f747005c2a0 con 0x7f74800732f0 2024-05-27T06:48:28.772 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:28.772 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:23.732532+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4287,4302]},"id":1}]} 2024-05-27T06:48:28.773 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 26 2024-05-27T06:48:28.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 msgr2=0x7f745c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:28.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 0x7f745c071940 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f748015d3d0 tx=0x7f7468008040 comp rx=0 tx=0).stop 2024-05-27T06:48:28.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 msgr2=0x7f74800bf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:28.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7470007e70 tx=0x7f747000e4b0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 shutdown_connections 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74801546f0 0x7f74800c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f745c06f430 0x7f745c071940 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74800c7510 0x7f74800cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 --2- 172.21.15.81:0/4206300340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800732f0 0x7f74800bf460 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 >> 172.21.15.81:0/4206300340 conn(0x7f7480070700 msgr2=0x7f74800c58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 shutdown_connections 2024-05-27T06:48:28.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:28.778+0000 7f7485600640 1 -- 172.21.15.81:0/4206300340 wait complete. 2024-05-27T06:48:28.793 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:29.674 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/30 file after the snapshot 2024-05-27T06:48:29.677 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/31 file after the snapshot 2024-05-27T06:48:29.679 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/32 file after the snapshot 2024-05-27T06:48:29.681 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/33 file after the snapshot 2024-05-27T06:48:29.684 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/34 file after the snapshot 2024-05-27T06:48:29.687 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/35 file after the snapshot 2024-05-27T06:48:29.689 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/36 file after the snapshot 2024-05-27T06:48:29.691 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/37 file after the snapshot 2024-05-27T06:48:29.694 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/38 file after the snapshot 2024-05-27T06:48:29.696 INFO:tasks.workunit.client.0.smithi081.stdout: create 101/39 file after the snapshot 2024-05-27T06:48:29.699 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/.snap/snap-subdir-test 2024-05-27T06:48:29.716 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/30 file after the snapshot 2024-05-27T06:48:29.719 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/31 file after the snapshot 2024-05-27T06:48:29.722 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/32 file after the snapshot 2024-05-27T06:48:29.724 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/33 file after the snapshot 2024-05-27T06:48:29.727 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/34 file after the snapshot 2024-05-27T06:48:29.729 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/35 file after the snapshot 2024-05-27T06:48:29.732 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/36 file after the snapshot 2024-05-27T06:48:29.734 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/37 file after the snapshot 2024-05-27T06:48:29.736 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/38 file after the snapshot 2024-05-27T06:48:29.738 INFO:tasks.workunit.client.0.smithi081.stdout: create 102/39 file after the snapshot 2024-05-27T06:48:29.741 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/.snap/snap-subdir-test 2024-05-27T06:48:30.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:30.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.962+0000 7fe653200640 1 Processor -- start 2024-05-27T06:48:30.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.962+0000 7fe653200640 1 -- start start 2024-05-27T06:48:30.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe653200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 0x7fe64c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:30.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe653200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c072840 con 0x7fe64c161cd0 2024-05-27T06:48:30.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe653200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c0c5270 con 0x7fe64c072430 2024-05-27T06:48:30.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe653200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c0c5040 con 0x7fe64c0c5b60 2024-05-27T06:48:30.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe651200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe64c161cd0 0x7fe64c1640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46248/0 (socket says 172.21.15.81:46248) 2024-05-27T06:48:30.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe651200640 1 -- 172.21.15.81:0/3044164379 learned_addr learned my addr 172.21.15.81:0/3044164379 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:30.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe643400640 1 --2- 172.21.15.81:0/3044164379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 0x7fe64c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391604345 0 0) 0x7fe64c072840 con 0x7fe64c161cd0 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe630003530 con 0x7fe64c161cd0 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881954349 0 0) 0x7fe64c0c5270 con 0x7fe64c072430 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64c072840 con 0x7fe64c072430 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe643400640 1 -- 172.21.15.81:0/3044164379 >> v1:172.21.15.106:6789/0 conn(0x7fe64c072430 legacy=0x7fe64c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe643400640 1 -- 172.21.15.81:0/3044164379 >> v1:172.21.15.81:6789/0 conn(0x7fe64c161cd0 legacy=0x7fe64c1640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:30.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.966+0000 7fe643400640 1 -- 172.21.15.81:0/3044164379 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c0c7810 con 0x7fe64c0c5b60 2024-05-27T06:48:30.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe643400640 1 --2- 172.21.15.81:0/3044164379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 0x7fe64c0c5f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe634009dc0 tx=0x7fe63401c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b99699c5b65f77a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:30.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340b2070 con 0x7fe64c0c5b60 2024-05-27T06:48:30.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6340a5850 con 0x7fe64c0c5b60 2024-05-27T06:48:30.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe64b400640 1 -- 172.21.15.81:0/3044164379 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6340a5c20 con 0x7fe64c0c5b60 2024-05-27T06:48:30.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 -- 172.21.15.81:0/3044164379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 msgr2=0x7fe64c0c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:30.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 --2- 172.21.15.81:0/3044164379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 0x7fe64c0c5f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe634009dc0 tx=0x7fe63401c990 comp rx=0 tx=0).stop 2024-05-27T06:48:30.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 -- 172.21.15.81:0/3044164379 shutdown_connections 2024-05-27T06:48:30.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 --2- 172.21.15.81:0/3044164379 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c0c5b60 0x7fe64c0c5f70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:30.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 -- 172.21.15.81:0/3044164379 >> 172.21.15.81:0/3044164379 conn(0x7fe64c070700 msgr2=0x7fe64c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:30.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 -- 172.21.15.81:0/3044164379 shutdown_connections 2024-05-27T06:48:30.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.970+0000 7fe653200640 1 -- 172.21.15.81:0/3044164379 wait complete. 2024-05-27T06:48:30.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 Processor -- start 2024-05-27T06:48:30.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- start start 2024-05-27T06:48:30.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:30.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 0x7fe64c074940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:30.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 0x7fe64c149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:30.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c0c7dd0 con 0x7fe64c072430 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c0c75c0 con 0x7fe64c0c5b60 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c0c5610 con 0x7fe64c161cd0 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42094/0 (socket says 172.21.15.81:42094) 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 -- 172.21.15.81:0/2785914937 learned_addr learned my addr 172.21.15.81:0/2785914937 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe651200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 0x7fe64c149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe643400640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 0x7fe64c074940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 -- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 msgr2=0x7fe64c149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 0x7fe64c149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 -- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 msgr2=0x7fe64c074940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 0x7fe64c074940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 -- 172.21.15.81:0/2785914937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c14a890 con 0x7fe64c072430 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe651200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 0x7fe64c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:48:30.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe643400640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 0x7fe64c074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe64be00640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00c430 tx=0x7fe63c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c017070 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c0052f0 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c155ca0 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c003840 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.974+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64c156260 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe64c14aa60 con 0x7fe64c072430 2024-05-27T06:48:30.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe63c01a070 con 0x7fe64c072430 2024-05-27T06:48:30.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe648c00640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 0x7fe628071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:30.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fe63c08e820 con 0x7fe64c072430 2024-05-27T06:48:30.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe643400640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 0x7fe628071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:30.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.978+0000 7fe643400640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 0x7fe628071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe6340af000 tx=0x7fe6340a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:30.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:30.982+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe63c05b550 con 0x7fe64c072430 2024-05-27T06:48:31.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.170+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe64c06a560 con 0x7fe64c072430 2024-05-27T06:48:31.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe648c00640 1 -- 172.21.15.81:0/2785914937 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6074 (secure 0 0 0) 0x7fe63c05ae50 con 0x7fe64c072430 2024-05-27T06:48:31.167 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:31.167 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:29.629593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:48:31.168 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 27 2024-05-27T06:48:31.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 msgr2=0x7fe628071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:31.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 0x7fe628071940 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe6340af000 tx=0x7fe6340a7070 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 msgr2=0x7fe64c14a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00c430 tx=0x7fe63c00c8f0 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 shutdown_connections 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe64c161cd0 0x7fe64c149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe62806f430 0x7fe628071940 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe64c0c5b60 0x7fe64c074940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 --2- 172.21.15.81:0/2785914937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe64c072430 0x7fe64c14a350 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 >> 172.21.15.81:0/2785914937 conn(0x7fe64c070700 msgr2=0x7fe64c075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 shutdown_connections 2024-05-27T06:48:31.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:31.174+0000 7fe653200640 1 -- 172.21.15.81:0/2785914937 wait complete. 2024-05-27T06:48:31.189 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:31.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:48:31.921 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:48:33.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.354+0000 7fa063600640 1 Processor -- start 2024-05-27T06:48:33.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.354+0000 7fa063600640 1 -- start start 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa063600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa063600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa05c161cd0 0x7fa05c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa063600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05c072840 con 0x7fa05c072430 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa063600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa05c0c5040 con 0x7fa05c161cd0 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa063600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c0c75c0 con 0x7fa05c0c5b60 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa061600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa05c161cd0 0x7fa05c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa060c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa05c072430 0x7fa05c159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46254/0 (socket says 172.21.15.81:46254) 2024-05-27T06:48:33.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa060c00640 1 -- 172.21.15.81:0/1621494609 learned_addr learned my addr 172.21.15.81:0/1621494609 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:33.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa05be00640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:33.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa061600640 1 -- 172.21.15.81:0/1621494609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 msgr2=0x7fa05c0c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:33.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa061600640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c0c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa061600640 1 -- 172.21.15.81:0/1621494609 >> v1:172.21.15.81:6789/0 conn(0x7fa05c072430 legacy=0x7fa05c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.358+0000 7fa061600640 1 -- 172.21.15.81:0/1621494609 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c0c5270 con 0x7fa05c161cd0 2024-05-27T06:48:33.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa061600640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c161cd0 0x7fa05c1640f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097c0 tx=0x7fa04c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f0a0aaadc86a3d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:33.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa05b400640 1 -- 172.21.15.81:0/1621494609 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0b0070 con 0x7fa05c161cd0 2024-05-27T06:48:33.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa05b400640 1 -- 172.21.15.81:0/1621494609 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04c0ab600 con 0x7fa05c161cd0 2024-05-27T06:48:33.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa05b400640 1 -- 172.21.15.81:0/1621494609 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0ab9d0 con 0x7fa05c161cd0 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- 172.21.15.81:0/1621494609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c161cd0 msgr2=0x7fa05c1640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c161cd0 0x7fa05c1640f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097c0 tx=0x7fa04c01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- 172.21.15.81:0/1621494609 shutdown_connections 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c161cd0 0x7fa05c1640f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 --2- 172.21.15.81:0/1621494609 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- 172.21.15.81:0/1621494609 >> 172.21.15.81:0/1621494609 conn(0x7fa05c070700 msgr2=0x7fa05c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- 172.21.15.81:0/1621494609 shutdown_connections 2024-05-27T06:48:33.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- 172.21.15.81:0/1621494609 wait complete. 2024-05-27T06:48:33.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 Processor -- start 2024-05-27T06:48:33.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.362+0000 7fa063600640 1 -- start start 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa05c161cd0 0x7fa05c148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0ce6f0 con 0x7fa05c072430 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0c7dd0 con 0x7fa05c161cd0 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c0c5610 con 0x7fa05c0c5b60 2024-05-27T06:48:33.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42114/0 (socket says 172.21.15.81:42114) 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 -- 172.21.15.81:0/1488051154 learned_addr learned my addr 172.21.15.81:0/1488051154 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa05be00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 -- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 msgr2=0x7fa05c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 -- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa05c161cd0 msgr2=0x7fa05c148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa05c161cd0 0x7fa05c148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 -- 172.21.15.81:0/1488051154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c144ab0 con 0x7fa05c072430 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa05be00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa060c00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa05400cb70 tx=0x7fa05400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054017070 con 0x7fa05c072430 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05400ad00 con 0x7fa05c072430 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05c1f6370 con 0x7fa05c072430 2024-05-27T06:48:33.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054012640 con 0x7fa05c072430 2024-05-27T06:48:33.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05c1f6930 con 0x7fa05c072430 2024-05-27T06:48:33.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.366+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa05c06a560 con 0x7fa05c072430 2024-05-27T06:48:33.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.370+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa05401a070 con 0x7fa05c072430 2024-05-27T06:48:33.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.370+0000 7fa058c00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 0x7fa03c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:33.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.370+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fa05408efe0 con 0x7fa05c072430 2024-05-27T06:48:33.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.370+0000 7fa05be00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 0x7fa03c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:33.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.370+0000 7fa05be00640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 0x7fa03c071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c8a60 tx=0x7fa048008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:33.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.374+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa05405be90 con 0x7fa05c072430 2024-05-27T06:48:33.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.562+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa05c0c5940 con 0x7fa05c072430 2024-05-27T06:48:33.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa058c00640 1 -- 172.21.15.81:0/1488051154 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6074 (secure 0 0 0) 0x7fa05405b790 con 0x7fa05c072430 2024-05-27T06:48:33.560 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:33.560 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:29.629593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4285,4281]},"id":1}]} 2024-05-27T06:48:33.560 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 27 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 msgr2=0x7fa03c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 0x7fa03c071910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c8a60 tx=0x7fa048008040 comp rx=0 tx=0).stop 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 msgr2=0x7fa05c1620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa05400cb70 tx=0x7fa05400a430 comp rx=0 tx=0).stop 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 shutdown_connections 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa05c0c5b60 0x7fa05c144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa03c06f400 0x7fa03c071910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa05c161cd0 0x7fa05c148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 --2- 172.21.15.81:0/1488051154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa05c072430 0x7fa05c1620e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:33.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 >> 172.21.15.81:0/1488051154 conn(0x7fa05c070700 msgr2=0x7fa05c147410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:33.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 shutdown_connections 2024-05-27T06:48:33.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:33.566+0000 7fa063600640 1 -- 172.21.15.81:0/1488051154 wait complete. 2024-05-27T06:48:33.582 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:48:34.845 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/30 file after the snapshot 2024-05-27T06:48:34.847 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/31 file after the snapshot 2024-05-27T06:48:34.850 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/32 file after the snapshot 2024-05-27T06:48:34.853 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/33 file after the snapshot 2024-05-27T06:48:34.856 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/34 file after the snapshot 2024-05-27T06:48:34.858 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/35 file after the snapshot 2024-05-27T06:48:34.860 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/36 file after the snapshot 2024-05-27T06:48:34.863 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/37 file after the snapshot 2024-05-27T06:48:34.866 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/38 file after the snapshot 2024-05-27T06:48:34.869 INFO:tasks.workunit.client.0.smithi081.stdout: create 103/39 file after the snapshot 2024-05-27T06:48:34.872 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/.snap/snap-subdir-test 2024-05-27T06:48:35.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:35.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 Processor -- start 2024-05-27T06:48:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 -- start start 2024-05-27T06:48:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f021806c790 0x7f021806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f021815c480 con 0x7f02181564a0 2024-05-27T06:48:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f02180c5bc0 con 0x7f021806c790 2024-05-27T06:48:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f021815c720 con 0x7f021815ff90 2024-05-27T06:48:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021e800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f021806c790 0x7f021806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021e800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:41672/0 (socket says 172.21.15.81:41672) 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021e800640 1 -- 172.21.15.81:0/13107844 learned_addr learned my addr 172.21.15.81:0/13107844 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021d400640 1 -- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 msgr2=0x7f02181603a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021d400640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021d400640 1 -- 172.21.15.81:0/13107844 >> v1:172.21.15.106:6789/0 conn(0x7f02181564a0 legacy=0x7f02180c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021d400640 1 -- 172.21.15.81:0/13107844 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021815ca00 con 0x7f021806c790 2024-05-27T06:48:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.750+0000 7f021e800640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021d400640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021806c790 0x7f021806cba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f02080097c0 tx=0x7f020801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b763068eaf2245c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021ca00640 1 -- 172.21.15.81:0/13107844 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02080b0070 con 0x7f021806c790 2024-05-27T06:48:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021ca00640 1 -- 172.21.15.81:0/13107844 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02080ab600 con 0x7f021806c790 2024-05-27T06:48:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021ca00640 1 -- 172.21.15.81:0/13107844 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02080ab9d0 con 0x7f021806c790 2024-05-27T06:48:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 -- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021806c790 msgr2=0x7f021806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021806c790 0x7f021806cba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f02080097c0 tx=0x7f020801c4a0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 -- 172.21.15.81:0/13107844 shutdown_connections 2024-05-27T06:48:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f021815ff90 0x7f02181603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 --2- 172.21.15.81:0/13107844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021806c790 0x7f021806cba0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 -- 172.21.15.81:0/13107844 >> 172.21.15.81:0/13107844 conn(0x7f021806dea0 msgr2=0x7f02180cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 -- 172.21.15.81:0/13107844 shutdown_connections 2024-05-27T06:48:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.754+0000 7f021f200640 1 -- 172.21.15.81:0/13107844 wait complete. 2024-05-27T06:48:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 Processor -- start 2024-05-27T06:48:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 -- start start 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02181564a0 0x7f021814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021815ff90 0x7f0218141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f02181608e0 con 0x7f021806c790 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f021815fa70 con 0x7f021815ff90 2024-05-27T06:48:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f02181568b0 con 0x7f02181564a0 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36116/0 (socket says 172.21.15.81:36116) 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 -- 172.21.15.81:0/285291089 learned_addr learned my addr 172.21.15.81:0/285291089 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021d400640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02181564a0 0x7f021814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021e800640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021815ff90 0x7f0218141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 -- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02181564a0 msgr2=0x7f021814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02181564a0 0x7f021814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 -- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021815ff90 msgr2=0x7f0218141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021815ff90 0x7f0218141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 -- 172.21.15.81:0/285291089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02181603a0 con 0x7f021806c790 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.758+0000 7f021de00640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f020c001ec0 tx=0x7f020c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c017070 con 0x7f021806c790 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c0052d0 con 0x7f021806c790 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c003840 con 0x7f021806c790 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021814e310 con 0x7f021806c790 2024-05-27T06:48:35.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f021814e8d0 con 0x7f021806c790 2024-05-27T06:48:35.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0218050870 con 0x7f021806c790 2024-05-27T06:48:35.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f020c01a070 con 0x7f021806c790 2024-05-27T06:48:35.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 0x7f01fc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:35.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.762+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f020c08e820 con 0x7f021806c790 2024-05-27T06:48:35.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.766+0000 7f021d400640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 0x7f01fc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:35.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.766+0000 7f021d400640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 0x7f01fc071940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f02080ad000 tx=0x7f0208008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:35.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.770+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f020c005480 con 0x7f021806c790 2024-05-27T06:48:35.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.958+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0218154770 con 0x7f021806c790 2024-05-27T06:48:35.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f0212000640 1 -- 172.21.15.81:0/285291089 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6079 (secure 0 0 0) 0x7f020c05b550 con 0x7f021806c790 2024-05-27T06:48:35.956 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:35.956 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:48:35.957 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2024-05-27T06:48:35.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 msgr2=0x7f01fc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 0x7f01fc071940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f02080ad000 tx=0x7f0208008040 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 msgr2=0x7f021814f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f020c001ec0 tx=0x7f020c00c890 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 shutdown_connections 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f02181564a0 0x7f021814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f01fc06f430 0x7f01fc071940 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f021815ff90 0x7f0218141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 --2- 172.21.15.81:0/285291089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f021806c790 0x7f021814f520 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 >> 172.21.15.81:0/285291089 conn(0x7f021806dea0 msgr2=0x7f021814bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 shutdown_connections 2024-05-27T06:48:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:35.962+0000 7f021f200640 1 -- 172.21.15.81:0/285291089 wait complete. 2024-05-27T06:48:35.977 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=4235 2024-05-27T06:48:35.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2024-05-27T06:48:39.897 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/30 file after the snapshot 2024-05-27T06:48:39.900 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/31 file after the snapshot 2024-05-27T06:48:39.902 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/32 file after the snapshot 2024-05-27T06:48:39.905 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/33 file after the snapshot 2024-05-27T06:48:39.908 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/34 file after the snapshot 2024-05-27T06:48:39.910 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/35 file after the snapshot 2024-05-27T06:48:39.913 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/36 file after the snapshot 2024-05-27T06:48:39.916 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/37 file after the snapshot 2024-05-27T06:48:39.919 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/38 file after the snapshot 2024-05-27T06:48:39.922 INFO:tasks.workunit.client.0.smithi081.stdout: create 104/39 file after the snapshot 2024-05-27T06:48:39.925 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/.snap/snap-subdir-test 2024-05-27T06:48:39.967 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/30 file after the snapshot 2024-05-27T06:48:39.970 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/31 file after the snapshot 2024-05-27T06:48:39.972 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/32 file after the snapshot 2024-05-27T06:48:39.975 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/33 file after the snapshot 2024-05-27T06:48:39.977 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/34 file after the snapshot 2024-05-27T06:48:39.977 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-05-27T06:48:39.977 INFO:tasks.ceph.mds.e:Restarting daemon 2024-05-27T06:48:39.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-05-27T06:48:39.980 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/35 file after the snapshot 2024-05-27T06:48:39.983 INFO:tasks.ceph.mds.e:Started 2024-05-27T06:48:39.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:39.983 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/36 file after the snapshot 2024-05-27T06:48:39.985 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/37 file after the snapshot 2024-05-27T06:48:39.988 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/38 file after the snapshot 2024-05-27T06:48:39.990 INFO:tasks.workunit.client.0.smithi081.stdout: create 105/39 file after the snapshot 2024-05-27T06:48:39.994 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/.snap/snap-subdir-test 2024-05-27T06:48:40.012 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/30 file after the snapshot 2024-05-27T06:48:40.015 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/31 file after the snapshot 2024-05-27T06:48:40.018 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/32 file after the snapshot 2024-05-27T06:48:40.021 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/33 file after the snapshot 2024-05-27T06:48:40.024 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/34 file after the snapshot 2024-05-27T06:48:40.026 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/35 file after the snapshot 2024-05-27T06:48:40.029 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/36 file after the snapshot 2024-05-27T06:48:40.031 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/37 file after the snapshot 2024-05-27T06:48:40.034 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/38 file after the snapshot 2024-05-27T06:48:40.036 INFO:tasks.workunit.client.0.smithi081.stdout: create 106/39 file after the snapshot 2024-05-27T06:48:40.038 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/.snap/snap-subdir-test 2024-05-27T06:48:40.055 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/30 file after the snapshot 2024-05-27T06:48:40.057 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/31 file after the snapshot 2024-05-27T06:48:40.059 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/32 file after the snapshot 2024-05-27T06:48:40.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/33 file after the snapshot 2024-05-27T06:48:40.064 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/34 file after the snapshot 2024-05-27T06:48:40.067 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/35 file after the snapshot 2024-05-27T06:48:40.069 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/36 file after the snapshot 2024-05-27T06:48:40.072 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/37 file after the snapshot 2024-05-27T06:48:40.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/38 file after the snapshot 2024-05-27T06:48:40.077 INFO:tasks.workunit.client.0.smithi081.stdout: create 107/39 file after the snapshot 2024-05-27T06:48:40.080 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/.snap/snap-subdir-test 2024-05-27T06:48:40.100 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/30 file after the snapshot 2024-05-27T06:48:40.101 INFO:tasks.ceph.mds.e.smithi081.stdout:starting mds.e at 2024-05-27T06:48:40.102 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/31 file after the snapshot 2024-05-27T06:48:40.105 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/32 file after the snapshot 2024-05-27T06:48:40.108 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/33 file after the snapshot 2024-05-27T06:48:40.110 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/34 file after the snapshot 2024-05-27T06:48:40.112 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/35 file after the snapshot 2024-05-27T06:48:40.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/36 file after the snapshot 2024-05-27T06:48:40.116 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/37 file after the snapshot 2024-05-27T06:48:40.119 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/38 file after the snapshot 2024-05-27T06:48:40.121 INFO:tasks.workunit.client.0.smithi081.stdout: create 108/39 file after the snapshot 2024-05-27T06:48:40.123 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/.snap/snap-subdir-test 2024-05-27T06:48:40.139 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/30 file after the snapshot 2024-05-27T06:48:40.141 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/31 file after the snapshot 2024-05-27T06:48:40.143 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/32 file after the snapshot 2024-05-27T06:48:40.146 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/33 file after the snapshot 2024-05-27T06:48:40.148 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/34 file after the snapshot 2024-05-27T06:48:40.150 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/35 file after the snapshot 2024-05-27T06:48:40.152 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/36 file after the snapshot 2024-05-27T06:48:40.156 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/37 file after the snapshot 2024-05-27T06:48:40.158 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/38 file after the snapshot 2024-05-27T06:48:40.160 INFO:tasks.workunit.client.0.smithi081.stdout: create 109/39 file after the snapshot 2024-05-27T06:48:40.162 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/.snap/snap-subdir-test 2024-05-27T06:48:40.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 Processor -- start 2024-05-27T06:48:40.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 -- start start 2024-05-27T06:48:40.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7208074540 0x7f7208161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7208162210 0x7f7208164630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7208074f80 con 0x7f7208074130 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f7208075260 con 0x7f7208074540 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7210200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7208073650 con 0x7f7208162210 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720d800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7208074130 0x7f720806d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35776/0 (socket says 172.21.15.81:35776) 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7208074540 0x7f7208161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720d800640 1 -- 172.21.15.81:0/4140083125 learned_addr learned my addr 172.21.15.81:0/4140083125 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:40.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720e200640 1 --2- 172.21.15.81:0/4140083125 >> v2:172.21.15.106:3300/0 conn(0x7f7208162210 0x7f7208164630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7207e00640 1 -- 172.21.15.81:0/4140083125 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330755266 0 0) 0x7f7208074f80 con 0x7f7208074130 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f7207e00640 1 -- 172.21.15.81:0/4140083125 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71f0005740 con 0x7f7208074130 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720ce00640 1 -- 172.21.15.81:0/4140083125 >> v2:172.21.15.106:3300/0 conn(0x7f7208162210 msgr2=0x7f7208164630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720ce00640 1 --2- 172.21.15.81:0/4140083125 >> v2:172.21.15.106:3300/0 conn(0x7f7208162210 0x7f7208164630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720ce00640 1 -- 172.21.15.81:0/4140083125 >> v1:172.21.15.81:6789/0 conn(0x7f7208074130 legacy=0x7f720806d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.173+0000 7f720ce00640 1 -- 172.21.15.81:0/4140083125 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7208073930 con 0x7f7208074540 2024-05-27T06:48:40.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f720ce00640 1 --2- 172.21.15.81:0/4140083125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208074540 0x7f7208161cd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7208073540 tx=0x7f71f4020860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ce840173415e73 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:40.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7207e00640 1 -- 172.21.15.81:0/4140083125 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f40aa6f0 con 0x7f7208074540 2024-05-27T06:48:40.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7207e00640 1 -- 172.21.15.81:0/4140083125 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f40aa890 con 0x7f7208074540 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7207e00640 1 -- 172.21.15.81:0/4140083125 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f40aac60 con 0x7f7208074540 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 -- 172.21.15.81:0/4140083125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208074540 msgr2=0x7f7208161cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 --2- 172.21.15.81:0/4140083125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208074540 0x7f7208161cd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7208073540 tx=0x7f71f4020860 comp rx=0 tx=0).stop 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 -- 172.21.15.81:0/4140083125 shutdown_connections 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 --2- 172.21.15.81:0/4140083125 >> v2:172.21.15.106:3300/0 conn(0x7f7208162210 0x7f7208164630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 --2- 172.21.15.81:0/4140083125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208074540 0x7f7208161cd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 -- 172.21.15.81:0/4140083125 >> 172.21.15.81:0/4140083125 conn(0x7f7208071770 msgr2=0x7f72080ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:40.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 -- 172.21.15.81:0/4140083125 shutdown_connections 2024-05-27T06:48:40.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.177+0000 7f7210200640 1 -- 172.21.15.81:0/4140083125 wait complete. 2024-05-27T06:48:40.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 Processor -- start 2024-05-27T06:48:40.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 -- start start 2024-05-27T06:48:40.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 0x7f72080cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208162210 0x7f72081497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f720815a2e0 con 0x7f7208162210 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72080c7570 con 0x7f7208074540 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f7210200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7208074bd0 con 0x7f7208074130 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f720d800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f720ce00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 0x7f72080cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f720d800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:41690/0 (socket says 172.21.15.81:41690) 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f720d800640 1 -- 172.21.15.81:0/2771559090 learned_addr learned my addr 172.21.15.81:0/2771559090 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:40.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.181+0000 7f720e200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208162210 0x7f72081497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 -- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 msgr2=0x7f72080cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 0x7f72080cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 -- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208162210 msgr2=0x7f72081497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208162210 0x7f72081497a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 -- 172.21.15.81:0/2771559090 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72080cd680 con 0x7f7208074130 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720ce00640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 0x7f72080cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f720d800640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c400 tx=0x7f71fc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc017070 con 0x7f7208074130 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71fc005cf0 con 0x7f7208074130 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7208157c70 con 0x7f7208074130 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0072f0 con 0x7f7208074130 2024-05-27T06:48:40.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72081581f0 con 0x7f7208074130 2024-05-27T06:48:40.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720806a560 con 0x7f7208074130 2024-05-27T06:48:40.182 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/30 file after the snapshot 2024-05-27T06:48:40.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.185+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f71fc01a070 con 0x7f7208074130 2024-05-27T06:48:40.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.189+0000 7f7205600640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 0x7f71e8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:40.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.189+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7f71fc08e6e0 con 0x7f7208074130 2024-05-27T06:48:40.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.189+0000 7f720ce00640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 0x7f71e8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:40.184 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/31 file after the snapshot 2024-05-27T06:48:40.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.189+0000 7f720ce00640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 0x7f71e8071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71f40b3000 tx=0x7f71f40ab000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:40.186 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/32 file after the snapshot 2024-05-27T06:48:40.188 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/33 file after the snapshot 2024-05-27T06:48:40.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.193+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f71fc05ae50 con 0x7f7208074130 2024-05-27T06:48:40.191 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/34 file after the snapshot 2024-05-27T06:48:40.193 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/35 file after the snapshot 2024-05-27T06:48:40.196 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/36 file after the snapshot 2024-05-27T06:48:40.198 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/37 file after the snapshot 2024-05-27T06:48:40.200 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/38 file after the snapshot 2024-05-27T06:48:40.202 INFO:tasks.workunit.client.0.smithi081.stdout: create 110/39 file after the snapshot 2024-05-27T06:48:40.205 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/.snap/snap-subdir-test 2024-05-27T06:48:40.220 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/30 file after the snapshot 2024-05-27T06:48:40.222 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/31 file after the snapshot 2024-05-27T06:48:40.224 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/32 file after the snapshot 2024-05-27T06:48:40.226 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/33 file after the snapshot 2024-05-27T06:48:40.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/34 file after the snapshot 2024-05-27T06:48:40.231 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/35 file after the snapshot 2024-05-27T06:48:40.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/36 file after the snapshot 2024-05-27T06:48:40.235 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/37 file after the snapshot 2024-05-27T06:48:40.237 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/38 file after the snapshot 2024-05-27T06:48:40.239 INFO:tasks.workunit.client.0.smithi081.stdout: create 111/39 file after the snapshot 2024-05-27T06:48:40.241 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/.snap/snap-subdir-test 2024-05-27T06:48:40.257 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/30 file after the snapshot 2024-05-27T06:48:40.258 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/31 file after the snapshot 2024-05-27T06:48:40.261 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/32 file after the snapshot 2024-05-27T06:48:40.263 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/33 file after the snapshot 2024-05-27T06:48:40.265 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/34 file after the snapshot 2024-05-27T06:48:40.267 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/35 file after the snapshot 2024-05-27T06:48:40.270 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/36 file after the snapshot 2024-05-27T06:48:40.272 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/37 file after the snapshot 2024-05-27T06:48:40.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/38 file after the snapshot 2024-05-27T06:48:40.277 INFO:tasks.workunit.client.0.smithi081.stdout: create 112/39 file after the snapshot 2024-05-27T06:48:40.279 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/.snap/snap-subdir-test 2024-05-27T06:48:40.295 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/30 file after the snapshot 2024-05-27T06:48:40.297 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/31 file after the snapshot 2024-05-27T06:48:40.300 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/32 file after the snapshot 2024-05-27T06:48:40.302 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/33 file after the snapshot 2024-05-27T06:48:40.305 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/34 file after the snapshot 2024-05-27T06:48:40.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/35 file after the snapshot 2024-05-27T06:48:40.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/36 file after the snapshot 2024-05-27T06:48:40.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/37 file after the snapshot 2024-05-27T06:48:40.315 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/38 file after the snapshot 2024-05-27T06:48:40.317 INFO:tasks.workunit.client.0.smithi081.stdout: create 113/39 file after the snapshot 2024-05-27T06:48:40.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/.snap/snap-subdir-test 2024-05-27T06:48:40.338 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/30 file after the snapshot 2024-05-27T06:48:40.341 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/31 file after the snapshot 2024-05-27T06:48:40.343 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/32 file after the snapshot 2024-05-27T06:48:40.345 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/33 file after the snapshot 2024-05-27T06:48:40.348 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/34 file after the snapshot 2024-05-27T06:48:40.350 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/35 file after the snapshot 2024-05-27T06:48:40.353 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/36 file after the snapshot 2024-05-27T06:48:40.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/37 file after the snapshot 2024-05-27T06:48:40.358 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/38 file after the snapshot 2024-05-27T06:48:40.360 INFO:tasks.workunit.client.0.smithi081.stdout: create 114/39 file after the snapshot 2024-05-27T06:48:40.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/.snap/snap-subdir-test 2024-05-27T06:48:40.379 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/30 file after the snapshot 2024-05-27T06:48:40.381 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/31 file after the snapshot 2024-05-27T06:48:40.383 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/32 file after the snapshot 2024-05-27T06:48:40.386 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/33 file after the snapshot 2024-05-27T06:48:40.388 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/34 file after the snapshot 2024-05-27T06:48:40.390 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/35 file after the snapshot 2024-05-27T06:48:40.393 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/36 file after the snapshot 2024-05-27T06:48:40.395 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/37 file after the snapshot 2024-05-27T06:48:40.397 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/38 file after the snapshot 2024-05-27T06:48:40.400 INFO:tasks.workunit.client.0.smithi081.stdout: create 115/39 file after the snapshot 2024-05-27T06:48:40.402 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/.snap/snap-subdir-test 2024-05-27T06:48:40.418 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/30 file after the snapshot 2024-05-27T06:48:40.419 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/31 file after the snapshot 2024-05-27T06:48:40.422 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/32 file after the snapshot 2024-05-27T06:48:40.425 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/33 file after the snapshot 2024-05-27T06:48:40.427 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/34 file after the snapshot 2024-05-27T06:48:40.430 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/35 file after the snapshot 2024-05-27T06:48:40.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72080cacf0 con 0x7f7208074130 2024-05-27T06:48:40.435 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/36 file after the snapshot 2024-05-27T06:48:40.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7205600640 1 -- 172.21.15.81:0/2771559090 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6079 (secure 0 0 0) 0x7f71fc05f150 con 0x7f7208074130 2024-05-27T06:48:40.435 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:40.436 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4281,4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 msgr2=0x7f71e8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 0x7f71e8071910 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71f40b3000 tx=0x7f71f40ab000 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 msgr2=0x7f72081504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c400 tx=0x7f71fc00c8c0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 shutdown_connections 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71e806f400 0x7f71e8071910 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7208162210 0x7f72081497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7208074540 0x7f72080cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 --2- 172.21.15.81:0/2771559090 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7208074130 0x7f72081504d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 >> 172.21.15.81:0/2771559090 conn(0x7f7208071770 msgr2=0x7f72081643a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 shutdown_connections 2024-05-27T06:48:40.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:40.437+0000 7f7210200640 1 -- 172.21.15.81:0/2771559090 wait complete. 2024-05-27T06:48:40.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/37 file after the snapshot 2024-05-27T06:48:40.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/38 file after the snapshot 2024-05-27T06:48:40.439 INFO:tasks.workunit.client.0.smithi081.stdout: create 116/39 file after the snapshot 2024-05-27T06:48:40.442 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/.snap/snap-subdir-test 2024-05-27T06:48:40.457 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-05-27T06:48:40.458 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/30 file after the snapshot 2024-05-27T06:48:40.460 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/31 file after the snapshot 2024-05-27T06:48:40.462 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/32 file after the snapshot 2024-05-27T06:48:40.465 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/33 file after the snapshot 2024-05-27T06:48:40.467 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/34 file after the snapshot 2024-05-27T06:48:40.469 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/35 file after the snapshot 2024-05-27T06:48:40.778 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/36 file after the snapshot 2024-05-27T06:48:41.238 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/37 file after the snapshot 2024-05-27T06:48:41.241 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/38 file after the snapshot 2024-05-27T06:48:41.244 INFO:tasks.workunit.client.0.smithi081.stdout: create 117/39 file after the snapshot 2024-05-27T06:48:41.246 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/.snap/snap-subdir-test 2024-05-27T06:48:41.266 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/30 file after the snapshot 2024-05-27T06:48:41.269 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/31 file after the snapshot 2024-05-27T06:48:41.272 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/32 file after the snapshot 2024-05-27T06:48:41.274 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/33 file after the snapshot 2024-05-27T06:48:41.276 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/34 file after the snapshot 2024-05-27T06:48:41.279 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/35 file after the snapshot 2024-05-27T06:48:41.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/36 file after the snapshot 2024-05-27T06:48:41.284 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/37 file after the snapshot 2024-05-27T06:48:41.287 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/38 file after the snapshot 2024-05-27T06:48:41.289 INFO:tasks.workunit.client.0.smithi081.stdout: create 118/39 file after the snapshot 2024-05-27T06:48:41.291 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/.snap/snap-subdir-test 2024-05-27T06:48:41.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/30 file after the snapshot 2024-05-27T06:48:41.309 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/31 file after the snapshot 2024-05-27T06:48:41.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/32 file after the snapshot 2024-05-27T06:48:41.314 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/33 file after the snapshot 2024-05-27T06:48:41.317 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/34 file after the snapshot 2024-05-27T06:48:41.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/35 file after the snapshot 2024-05-27T06:48:41.323 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/36 file after the snapshot 2024-05-27T06:48:41.325 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/37 file after the snapshot 2024-05-27T06:48:41.327 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/38 file after the snapshot 2024-05-27T06:48:41.330 INFO:tasks.workunit.client.0.smithi081.stdout: create 119/39 file after the snapshot 2024-05-27T06:48:41.333 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/.snap/snap-subdir-test 2024-05-27T06:48:41.348 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/30 file after the snapshot 2024-05-27T06:48:41.350 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/31 file after the snapshot 2024-05-27T06:48:41.352 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/32 file after the snapshot 2024-05-27T06:48:41.354 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/33 file after the snapshot 2024-05-27T06:48:41.357 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/34 file after the snapshot 2024-05-27T06:48:41.360 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/35 file after the snapshot 2024-05-27T06:48:41.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/36 file after the snapshot 2024-05-27T06:48:41.364 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/37 file after the snapshot 2024-05-27T06:48:41.367 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/38 file after the snapshot 2024-05-27T06:48:41.369 INFO:tasks.workunit.client.0.smithi081.stdout: create 120/39 file after the snapshot 2024-05-27T06:48:41.371 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/.snap/snap-subdir-test 2024-05-27T06:48:41.387 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/30 file after the snapshot 2024-05-27T06:48:41.389 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/31 file after the snapshot 2024-05-27T06:48:41.392 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/32 file after the snapshot 2024-05-27T06:48:41.394 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/33 file after the snapshot 2024-05-27T06:48:41.397 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/34 file after the snapshot 2024-05-27T06:48:41.400 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/35 file after the snapshot 2024-05-27T06:48:41.402 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/36 file after the snapshot 2024-05-27T06:48:41.404 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/37 file after the snapshot 2024-05-27T06:48:41.406 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/38 file after the snapshot 2024-05-27T06:48:41.408 INFO:tasks.workunit.client.0.smithi081.stdout: create 121/39 file after the snapshot 2024-05-27T06:48:41.411 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/.snap/snap-subdir-test 2024-05-27T06:48:41.426 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/30 file after the snapshot 2024-05-27T06:48:41.428 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/31 file after the snapshot 2024-05-27T06:48:41.431 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/32 file after the snapshot 2024-05-27T06:48:41.433 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/33 file after the snapshot 2024-05-27T06:48:41.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/34 file after the snapshot 2024-05-27T06:48:41.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/35 file after the snapshot 2024-05-27T06:48:41.440 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/36 file after the snapshot 2024-05-27T06:48:41.443 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/37 file after the snapshot 2024-05-27T06:48:41.445 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/38 file after the snapshot 2024-05-27T06:48:41.448 INFO:tasks.workunit.client.0.smithi081.stdout: create 122/39 file after the snapshot 2024-05-27T06:48:41.450 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/.snap/snap-subdir-test 2024-05-27T06:48:41.465 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/30 file after the snapshot 2024-05-27T06:48:41.467 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/31 file after the snapshot 2024-05-27T06:48:41.470 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/32 file after the snapshot 2024-05-27T06:48:41.472 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/33 file after the snapshot 2024-05-27T06:48:41.475 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/34 file after the snapshot 2024-05-27T06:48:41.477 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/35 file after the snapshot 2024-05-27T06:48:41.479 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/36 file after the snapshot 2024-05-27T06:48:41.482 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/37 file after the snapshot 2024-05-27T06:48:41.485 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/38 file after the snapshot 2024-05-27T06:48:41.487 INFO:tasks.workunit.client.0.smithi081.stdout: create 123/39 file after the snapshot 2024-05-27T06:48:41.490 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/.snap/snap-subdir-test 2024-05-27T06:48:41.505 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/30 file after the snapshot 2024-05-27T06:48:41.507 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/31 file after the snapshot 2024-05-27T06:48:41.509 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/32 file after the snapshot 2024-05-27T06:48:41.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/33 file after the snapshot 2024-05-27T06:48:41.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/34 file after the snapshot 2024-05-27T06:48:41.516 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/35 file after the snapshot 2024-05-27T06:48:41.518 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/36 file after the snapshot 2024-05-27T06:48:41.521 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/37 file after the snapshot 2024-05-27T06:48:41.523 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/38 file after the snapshot 2024-05-27T06:48:41.525 INFO:tasks.workunit.client.0.smithi081.stdout: create 124/39 file after the snapshot 2024-05-27T06:48:41.527 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/.snap/snap-subdir-test 2024-05-27T06:48:41.543 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/30 file after the snapshot 2024-05-27T06:48:41.545 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/31 file after the snapshot 2024-05-27T06:48:41.547 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/32 file after the snapshot 2024-05-27T06:48:41.549 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/33 file after the snapshot 2024-05-27T06:48:41.551 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/34 file after the snapshot 2024-05-27T06:48:41.554 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/35 file after the snapshot 2024-05-27T06:48:41.556 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/36 file after the snapshot 2024-05-27T06:48:41.558 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/37 file after the snapshot 2024-05-27T06:48:41.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/38 file after the snapshot 2024-05-27T06:48:41.563 INFO:tasks.workunit.client.0.smithi081.stdout: create 125/39 file after the snapshot 2024-05-27T06:48:41.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/.snap/snap-subdir-test 2024-05-27T06:48:41.581 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/30 file after the snapshot 2024-05-27T06:48:41.584 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/31 file after the snapshot 2024-05-27T06:48:41.586 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/32 file after the snapshot 2024-05-27T06:48:41.588 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/33 file after the snapshot 2024-05-27T06:48:41.590 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/34 file after the snapshot 2024-05-27T06:48:41.593 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/35 file after the snapshot 2024-05-27T06:48:41.595 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/36 file after the snapshot 2024-05-27T06:48:41.597 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/37 file after the snapshot 2024-05-27T06:48:41.599 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/38 file after the snapshot 2024-05-27T06:48:41.601 INFO:tasks.workunit.client.0.smithi081.stdout: create 126/39 file after the snapshot 2024-05-27T06:48:41.603 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/.snap/snap-subdir-test 2024-05-27T06:48:41.619 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/30 file after the snapshot 2024-05-27T06:48:41.621 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/31 file after the snapshot 2024-05-27T06:48:41.624 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/32 file after the snapshot 2024-05-27T06:48:41.626 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/33 file after the snapshot 2024-05-27T06:48:41.629 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/34 file after the snapshot 2024-05-27T06:48:41.631 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/35 file after the snapshot 2024-05-27T06:48:41.634 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/36 file after the snapshot 2024-05-27T06:48:41.636 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/37 file after the snapshot 2024-05-27T06:48:41.638 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/38 file after the snapshot 2024-05-27T06:48:41.641 INFO:tasks.workunit.client.0.smithi081.stdout: create 127/39 file after the snapshot 2024-05-27T06:48:41.643 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/.snap/snap-subdir-test 2024-05-27T06:48:41.658 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/30 file after the snapshot 2024-05-27T06:48:41.661 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/31 file after the snapshot 2024-05-27T06:48:41.663 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/32 file after the snapshot 2024-05-27T06:48:41.666 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/33 file after the snapshot 2024-05-27T06:48:41.668 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/34 file after the snapshot 2024-05-27T06:48:41.670 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/35 file after the snapshot 2024-05-27T06:48:41.673 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/36 file after the snapshot 2024-05-27T06:48:41.675 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/37 file after the snapshot 2024-05-27T06:48:41.677 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/38 file after the snapshot 2024-05-27T06:48:41.680 INFO:tasks.workunit.client.0.smithi081.stdout: create 128/39 file after the snapshot 2024-05-27T06:48:41.682 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/.snap/snap-subdir-test 2024-05-27T06:48:42.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:48:42.533 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/30 file after the snapshot 2024-05-27T06:48:42.536 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/31 file after the snapshot 2024-05-27T06:48:42.539 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/32 file after the snapshot 2024-05-27T06:48:42.542 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/33 file after the snapshot 2024-05-27T06:48:42.545 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/34 file after the snapshot 2024-05-27T06:48:42.548 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/35 file after the snapshot 2024-05-27T06:48:42.551 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/36 file after the snapshot 2024-05-27T06:48:42.553 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/37 file after the snapshot 2024-05-27T06:48:42.556 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/38 file after the snapshot 2024-05-27T06:48:42.558 INFO:tasks.workunit.client.0.smithi081.stdout: create 129/39 file after the snapshot 2024-05-27T06:48:42.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/.snap/snap-subdir-test 2024-05-27T06:48:42.578 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/30 file after the snapshot 2024-05-27T06:48:42.580 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/31 file after the snapshot 2024-05-27T06:48:42.582 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/32 file after the snapshot 2024-05-27T06:48:42.585 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/33 file after the snapshot 2024-05-27T06:48:42.587 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/34 file after the snapshot 2024-05-27T06:48:42.589 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/35 file after the snapshot 2024-05-27T06:48:42.592 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/36 file after the snapshot 2024-05-27T06:48:42.594 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/37 file after the snapshot 2024-05-27T06:48:42.596 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/38 file after the snapshot 2024-05-27T06:48:42.598 INFO:tasks.workunit.client.0.smithi081.stdout: create 130/39 file after the snapshot 2024-05-27T06:48:42.601 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/.snap/snap-subdir-test 2024-05-27T06:48:42.616 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/30 file after the snapshot 2024-05-27T06:48:42.618 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/31 file after the snapshot 2024-05-27T06:48:42.621 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/32 file after the snapshot 2024-05-27T06:48:42.623 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/33 file after the snapshot 2024-05-27T06:48:42.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.629+0000 7fe6f5a00640 1 Processor -- start 2024-05-27T06:48:42.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.629+0000 7fe6f5a00640 1 -- start start 2024-05-27T06:48:42.626 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/34 file after the snapshot 2024-05-27T06:48:42.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6f5a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 0x7fe6f00ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6f5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 0x7fe6f0160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6f5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6f015c910 con 0x7fe6f0156760 2024-05-27T06:48:42.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6f5a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe6f0071680 con 0x7fe6f00ce010 2024-05-27T06:48:42.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6f5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f015fca0 con 0x7fe6f0160130 2024-05-27T06:48:42.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6eea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe6f0156760 0x7fe6f0070f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35788/0 (socket says 172.21.15.81:35788) 2024-05-27T06:48:42.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6eea00640 1 -- 172.21.15.81:0/1652767044 learned_addr learned my addr 172.21.15.81:0/1652767044 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:48:42.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ef400640 1 --2- 172.21.15.81:0/1652767044 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 0x7fe6f0160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ee000640 1 --2- 172.21.15.81:0/1652767044 >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 0x7fe6f00ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.629 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/35 file after the snapshot 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ed600640 1 -- 172.21.15.81:0/1652767044 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983857928 0 0) 0x7fe6f015c910 con 0x7fe6f0156760 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ed600640 1 -- 172.21.15.81:0/1652767044 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6d4003530 con 0x7fe6f0156760 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ef400640 1 -- 172.21.15.81:0/1652767044 >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 msgr2=0x7fe6f00ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ef400640 1 --2- 172.21.15.81:0/1652767044 >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 0x7fe6f00ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ef400640 1 -- 172.21.15.81:0/1652767044 >> v1:172.21.15.81:6789/0 conn(0x7fe6f0156760 legacy=0x7fe6f0070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ef400640 1 -- 172.21.15.81:0/1652767044 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f015cbf0 con 0x7fe6f0160130 2024-05-27T06:48:42.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.633+0000 7fe6ee000640 1 --2- 172.21.15.81:0/1652767044 >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 0x7fe6f00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:48:42.631 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/36 file after the snapshot 2024-05-27T06:48:42.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6ef400640 1 --2- 172.21.15.81:0/1652767044 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 0x7fe6f0160540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009d90 tx=0x7fe6e801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a2e3e2f9b319aeb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:42.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6ed600640 1 -- 172.21.15.81:0/1652767044 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80b4070 con 0x7fe6f0160130 2024-05-27T06:48:42.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6ed600640 1 -- 172.21.15.81:0/1652767044 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e80af600 con 0x7fe6f0160130 2024-05-27T06:48:42.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6ed600640 1 -- 172.21.15.81:0/1652767044 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80af9d0 con 0x7fe6f0160130 2024-05-27T06:48:42.633 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/37 file after the snapshot 2024-05-27T06:48:42.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 -- 172.21.15.81:0/1652767044 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 msgr2=0x7fe6f0160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/1652767044 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 0x7fe6f0160540 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009d90 tx=0x7fe6e801f150 comp rx=0 tx=0).stop 2024-05-27T06:48:42.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 -- 172.21.15.81:0/1652767044 shutdown_connections 2024-05-27T06:48:42.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/1652767044 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0160130 0x7fe6f0160540 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/1652767044 >> v2:172.21.15.106:3300/0 conn(0x7fe6f00ce010 0x7fe6f00ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 -- 172.21.15.81:0/1652767044 >> 172.21.15.81:0/1652767044 conn(0x7fe6f006ee80 msgr2=0x7fe6f00707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 -- 172.21.15.81:0/1652767044 shutdown_connections 2024-05-27T06:48:42.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.637+0000 7fe6f5a00640 1 -- 172.21.15.81:0/1652767044 wait complete. 2024-05-27T06:48:42.635 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/38 file after the snapshot 2024-05-27T06:48:42.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 Processor -- start 2024-05-27T06:48:42.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 -- start start 2024-05-27T06:48:42.638 INFO:tasks.workunit.client.0.smithi081.stdout: create 131/39 file after the snapshot 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 0x7fe6f014f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 0x7fe6f014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6f0160130 0x7fe6f01578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f015d490 con 0x7fe6f00ce010 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f015d1b0 con 0x7fe6f0160130 2024-05-27T06:48:42.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.641+0000 7fe6f5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe6f015c620 con 0x7fe6f0156760 2024-05-27T06:48:42.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 0x7fe6f014f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 0x7fe6f014f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36152/0 (socket says 172.21.15.81:36152) 2024-05-27T06:48:42.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6eea00640 1 -- 172.21.15.81:0/3185349865 learned_addr learned my addr 172.21.15.81:0/3185349865 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:48:42.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ef400640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6f0160130 0x7fe6f01578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 0x7fe6f014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 -- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6f0160130 msgr2=0x7fe6f01578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6f0160130 0x7fe6f01578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 -- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 msgr2=0x7fe6f014f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 0x7fe6f014f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 -- 172.21.15.81:0/3185349865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f01f3c10 con 0x7fe6f0156760 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6ee000640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 0x7fe6f014fd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e8b0 tx=0x7fe6dc00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:42.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc017070 con 0x7fe6f0156760 2024-05-27T06:48:42.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00c8e0 con 0x7fe6f0156760 2024-05-27T06:48:42.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00ccb0 con 0x7fe6f0156760 2024-05-27T06:48:42.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f0159c00 con 0x7fe6f0156760 2024-05-27T06:48:42.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f015a1c0 con 0x7fe6f0156760 2024-05-27T06:48:42.642 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/.snap/snap-subdir-test 2024-05-27T06:48:42.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.645+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f006a560 con 0x7fe6f0156760 2024-05-27T06:48:42.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.649+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe6dc01a070 con 0x7fe6f0156760 2024-05-27T06:48:42.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.649+0000 7fe6e6a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 0x7fe6d0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:48:42.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.649+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5997+0+0 (secure 0 0 0) 0x7fe6dc08f470 con 0x7fe6f0156760 2024-05-27T06:48:42.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.649+0000 7fe6eea00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 0x7fe6d0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:48:42.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.649+0000 7fe6eea00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 0x7fe6d0071910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0007fd0 tx=0x7fe6e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:48:42.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.653+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe6dc05bbf0 con 0x7fe6f0156760 2024-05-27T06:48:42.658 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/30 file after the snapshot 2024-05-27T06:48:42.660 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/31 file after the snapshot 2024-05-27T06:48:42.663 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/32 file after the snapshot 2024-05-27T06:48:42.665 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/33 file after the snapshot 2024-05-27T06:48:42.667 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/34 file after the snapshot 2024-05-27T06:48:42.669 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/35 file after the snapshot 2024-05-27T06:48:42.672 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/36 file after the snapshot 2024-05-27T06:48:42.674 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/37 file after the snapshot 2024-05-27T06:48:42.676 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/38 file after the snapshot 2024-05-27T06:48:42.679 INFO:tasks.workunit.client.0.smithi081.stdout: create 132/39 file after the snapshot 2024-05-27T06:48:42.681 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/.snap/snap-subdir-test 2024-05-27T06:48:42.698 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/30 file after the snapshot 2024-05-27T06:48:42.700 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/31 file after the snapshot 2024-05-27T06:48:42.703 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/32 file after the snapshot 2024-05-27T06:48:42.706 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/33 file after the snapshot 2024-05-27T06:48:42.708 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/34 file after the snapshot 2024-05-27T06:48:42.710 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/35 file after the snapshot 2024-05-27T06:48:42.713 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/36 file after the snapshot 2024-05-27T06:48:42.715 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/37 file after the snapshot 2024-05-27T06:48:42.717 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/38 file after the snapshot 2024-05-27T06:48:42.720 INFO:tasks.workunit.client.0.smithi081.stdout: create 133/39 file after the snapshot 2024-05-27T06:48:42.722 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/.snap/snap-subdir-test 2024-05-27T06:48:42.738 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/30 file after the snapshot 2024-05-27T06:48:42.740 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/31 file after the snapshot 2024-05-27T06:48:42.742 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/32 file after the snapshot 2024-05-27T06:48:42.745 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/33 file after the snapshot 2024-05-27T06:48:42.747 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/34 file after the snapshot 2024-05-27T06:48:42.749 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/35 file after the snapshot 2024-05-27T06:48:42.752 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/36 file after the snapshot 2024-05-27T06:48:42.754 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/37 file after the snapshot 2024-05-27T06:48:42.756 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/38 file after the snapshot 2024-05-27T06:48:42.758 INFO:tasks.workunit.client.0.smithi081.stdout: create 134/39 file after the snapshot 2024-05-27T06:48:42.761 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/.snap/snap-subdir-test 2024-05-27T06:48:42.777 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/30 file after the snapshot 2024-05-27T06:48:42.779 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/31 file after the snapshot 2024-05-27T06:48:42.782 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/32 file after the snapshot 2024-05-27T06:48:42.784 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/33 file after the snapshot 2024-05-27T06:48:42.786 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/34 file after the snapshot 2024-05-27T06:48:42.789 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/35 file after the snapshot 2024-05-27T06:48:42.791 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/36 file after the snapshot 2024-05-27T06:48:42.794 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/37 file after the snapshot 2024-05-27T06:48:42.796 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/38 file after the snapshot 2024-05-27T06:48:42.799 INFO:tasks.workunit.client.0.smithi081.stdout: create 135/39 file after the snapshot 2024-05-27T06:48:42.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6f0073470 con 0x7fe6f0156760 2024-05-27T06:48:42.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6e6a00640 1 -- 172.21.15.81:0/3185349865 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7fe6dc05fef0 con 0x7fe6f0156760 2024-05-27T06:48:42.873 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:48:42.874 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:48:42.874 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:48:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 msgr2=0x7fe6d0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 0x7fe6d0071910 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0007fd0 tx=0x7fe6e0009040 comp rx=0 tx=0).stop 2024-05-27T06:48:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 msgr2=0x7fe6f014fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:48:42.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.877+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 0x7fe6f014fd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00e8b0 tx=0x7fe6dc00ed70 comp rx=0 tx=0).stop 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 shutdown_connections 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6f0156760 0x7fe6f014fd10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe6d006f400 0x7fe6d0071910 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6f0160130 0x7fe6f01578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 --2- 172.21.15.81:0/3185349865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6f00ce010 0x7fe6f014f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 >> 172.21.15.81:0/3185349865 conn(0x7fe6f006ee80 msgr2=0x7fe6f0074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 shutdown_connections 2024-05-27T06:48:42.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:48:42.881+0000 7fe6f5a00640 1 -- 172.21.15.81:0/3185349865 wait complete. 2024-05-27T06:48:42.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-05-27T06:48:42.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2024-05-27T06:48:43.775 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/.snap/snap-subdir-test 2024-05-27T06:48:43.795 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/30 file after the snapshot 2024-05-27T06:48:43.797 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/31 file after the snapshot 2024-05-27T06:48:43.801 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/32 file after the snapshot 2024-05-27T06:48:43.803 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/33 file after the snapshot 2024-05-27T06:48:43.807 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/34 file after the snapshot 2024-05-27T06:48:43.810 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/35 file after the snapshot 2024-05-27T06:48:43.812 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/36 file after the snapshot 2024-05-27T06:48:43.815 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/37 file after the snapshot 2024-05-27T06:48:43.817 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/38 file after the snapshot 2024-05-27T06:48:43.820 INFO:tasks.workunit.client.0.smithi081.stdout: create 136/39 file after the snapshot 2024-05-27T06:48:43.822 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/.snap/snap-subdir-test 2024-05-27T06:48:43.839 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/30 file after the snapshot 2024-05-27T06:48:43.842 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/31 file after the snapshot 2024-05-27T06:48:43.845 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/32 file after the snapshot 2024-05-27T06:48:43.848 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/33 file after the snapshot 2024-05-27T06:48:43.850 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/34 file after the snapshot 2024-05-27T06:48:43.853 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/35 file after the snapshot 2024-05-27T06:48:43.856 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/36 file after the snapshot 2024-05-27T06:48:43.858 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/37 file after the snapshot 2024-05-27T06:48:43.861 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/38 file after the snapshot 2024-05-27T06:48:43.864 INFO:tasks.workunit.client.0.smithi081.stdout: create 137/39 file after the snapshot 2024-05-27T06:48:43.866 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/.snap/snap-subdir-test 2024-05-27T06:48:43.882 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/30 file after the snapshot 2024-05-27T06:48:43.885 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/31 file after the snapshot 2024-05-27T06:48:43.888 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/32 file after the snapshot 2024-05-27T06:48:43.890 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/33 file after the snapshot 2024-05-27T06:48:43.893 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/34 file after the snapshot 2024-05-27T06:48:43.895 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/35 file after the snapshot 2024-05-27T06:48:43.897 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/36 file after the snapshot 2024-05-27T06:48:43.899 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/37 file after the snapshot 2024-05-27T06:48:43.902 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/38 file after the snapshot 2024-05-27T06:48:43.904 INFO:tasks.workunit.client.0.smithi081.stdout: create 138/39 file after the snapshot 2024-05-27T06:48:43.906 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/.snap/snap-subdir-test 2024-05-27T06:48:44.704 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/30 file after the snapshot 2024-05-27T06:48:44.706 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/31 file after the snapshot 2024-05-27T06:48:44.709 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/32 file after the snapshot 2024-05-27T06:48:44.712 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/33 file after the snapshot 2024-05-27T06:48:44.715 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/34 file after the snapshot 2024-05-27T06:48:44.718 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/35 file after the snapshot 2024-05-27T06:48:44.721 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/36 file after the snapshot 2024-05-27T06:48:44.727 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/37 file after the snapshot 2024-05-27T06:48:44.727 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/38 file after the snapshot 2024-05-27T06:48:44.728 INFO:tasks.workunit.client.0.smithi081.stdout: create 139/39 file after the snapshot 2024-05-27T06:48:44.730 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/.snap/snap-subdir-test 2024-05-27T06:48:44.747 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/30 file after the snapshot 2024-05-27T06:48:44.749 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/31 file after the snapshot 2024-05-27T06:48:44.752 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/32 file after the snapshot 2024-05-27T06:48:44.754 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/33 file after the snapshot 2024-05-27T06:48:44.756 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/34 file after the snapshot 2024-05-27T06:48:44.759 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/35 file after the snapshot 2024-05-27T06:48:44.761 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/36 file after the snapshot 2024-05-27T06:48:44.764 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/37 file after the snapshot 2024-05-27T06:48:44.766 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/38 file after the snapshot 2024-05-27T06:48:44.769 INFO:tasks.workunit.client.0.smithi081.stdout: create 140/39 file after the snapshot 2024-05-27T06:48:44.771 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/.snap/snap-subdir-test 2024-05-27T06:48:44.788 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/30 file after the snapshot 2024-05-27T06:48:45.035 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/31 file after the snapshot 2024-05-27T06:48:45.038 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/32 file after the snapshot 2024-05-27T06:48:45.040 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/33 file after the snapshot 2024-05-27T06:48:45.043 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/34 file after the snapshot 2024-05-27T06:48:45.046 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/35 file after the snapshot 2024-05-27T06:48:45.048 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/36 file after the snapshot 2024-05-27T06:48:45.051 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/37 file after the snapshot 2024-05-27T06:48:45.053 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/38 file after the snapshot 2024-05-27T06:48:45.055 INFO:tasks.workunit.client.0.smithi081.stdout: create 141/39 file after the snapshot 2024-05-27T06:48:45.058 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/.snap/snap-subdir-test 2024-05-27T06:48:45.074 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/30 file after the snapshot 2024-05-27T06:48:45.076 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/31 file after the snapshot 2024-05-27T06:48:45.079 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/32 file after the snapshot 2024-05-27T06:48:45.081 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/33 file after the snapshot 2024-05-27T06:48:45.083 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/34 file after the snapshot 2024-05-27T06:48:45.085 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/35 file after the snapshot 2024-05-27T06:48:45.087 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/36 file after the snapshot 2024-05-27T06:48:45.090 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/37 file after the snapshot 2024-05-27T06:48:45.092 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/38 file after the snapshot 2024-05-27T06:48:45.094 INFO:tasks.workunit.client.0.smithi081.stdout: create 142/39 file after the snapshot 2024-05-27T06:48:45.096 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/.snap/snap-subdir-test 2024-05-27T06:48:45.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/30 file after the snapshot 2024-05-27T06:48:45.116 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/31 file after the snapshot 2024-05-27T06:48:45.118 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/32 file after the snapshot 2024-05-27T06:48:45.121 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/33 file after the snapshot 2024-05-27T06:48:45.123 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/34 file after the snapshot 2024-05-27T06:48:45.125 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/35 file after the snapshot 2024-05-27T06:48:45.127 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/36 file after the snapshot 2024-05-27T06:48:45.130 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/37 file after the snapshot 2024-05-27T06:48:45.132 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/38 file after the snapshot 2024-05-27T06:48:45.134 INFO:tasks.workunit.client.0.smithi081.stdout: create 143/39 file after the snapshot 2024-05-27T06:48:45.138 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/.snap/snap-subdir-test 2024-05-27T06:48:45.153 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/30 file after the snapshot 2024-05-27T06:48:45.155 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/31 file after the snapshot 2024-05-27T06:48:45.157 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/32 file after the snapshot 2024-05-27T06:48:45.159 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/33 file after the snapshot 2024-05-27T06:48:45.162 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/34 file after the snapshot 2024-05-27T06:48:45.164 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/35 file after the snapshot 2024-05-27T06:48:45.166 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/36 file after the snapshot 2024-05-27T06:48:45.169 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/37 file after the snapshot 2024-05-27T06:48:45.171 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/38 file after the snapshot 2024-05-27T06:48:45.173 INFO:tasks.workunit.client.0.smithi081.stdout: create 144/39 file after the snapshot 2024-05-27T06:48:45.176 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/.snap/snap-subdir-test 2024-05-27T06:48:45.191 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/30 file after the snapshot 2024-05-27T06:48:45.193 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/31 file after the snapshot 2024-05-27T06:48:45.196 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/32 file after the snapshot 2024-05-27T06:48:45.198 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/33 file after the snapshot 2024-05-27T06:48:45.201 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/34 file after the snapshot 2024-05-27T06:48:45.203 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/35 file after the snapshot 2024-05-27T06:48:45.206 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/36 file after the snapshot 2024-05-27T06:48:45.208 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/37 file after the snapshot 2024-05-27T06:48:45.210 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/38 file after the snapshot 2024-05-27T06:48:45.213 INFO:tasks.workunit.client.0.smithi081.stdout: create 145/39 file after the snapshot 2024-05-27T06:48:45.215 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/.snap/snap-subdir-test 2024-05-27T06:48:45.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/30 file after the snapshot 2024-05-27T06:48:45.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/31 file after the snapshot 2024-05-27T06:48:45.235 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/32 file after the snapshot 2024-05-27T06:48:45.237 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/33 file after the snapshot 2024-05-27T06:48:45.239 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/34 file after the snapshot 2024-05-27T06:48:45.241 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/35 file after the snapshot 2024-05-27T06:48:45.244 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/36 file after the snapshot 2024-05-27T06:48:45.246 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/37 file after the snapshot 2024-05-27T06:48:45.248 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/38 file after the snapshot 2024-05-27T06:48:45.251 INFO:tasks.workunit.client.0.smithi081.stdout: create 146/39 file after the snapshot 2024-05-27T06:48:45.253 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/.snap/snap-subdir-test 2024-05-27T06:48:45.269 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/30 file after the snapshot 2024-05-27T06:48:45.271 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/31 file after the snapshot 2024-05-27T06:48:45.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/32 file after the snapshot 2024-05-27T06:48:45.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/33 file after the snapshot 2024-05-27T06:48:45.278 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/34 file after the snapshot 2024-05-27T06:48:45.280 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/35 file after the snapshot 2024-05-27T06:48:45.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/36 file after the snapshot 2024-05-27T06:48:45.284 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/37 file after the snapshot 2024-05-27T06:48:45.287 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/38 file after the snapshot 2024-05-27T06:48:45.290 INFO:tasks.workunit.client.0.smithi081.stdout: create 147/39 file after the snapshot 2024-05-27T06:48:45.292 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/.snap/snap-subdir-test 2024-05-27T06:48:45.308 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/30 file after the snapshot 2024-05-27T06:48:45.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/31 file after the snapshot 2024-05-27T06:48:45.313 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/32 file after the snapshot 2024-05-27T06:48:45.315 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/33 file after the snapshot 2024-05-27T06:48:45.317 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/34 file after the snapshot 2024-05-27T06:48:45.319 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/35 file after the snapshot 2024-05-27T06:48:45.322 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/36 file after the snapshot 2024-05-27T06:48:45.324 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/37 file after the snapshot 2024-05-27T06:48:45.326 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/38 file after the snapshot 2024-05-27T06:48:45.329 INFO:tasks.workunit.client.0.smithi081.stdout: create 148/39 file after the snapshot 2024-05-27T06:48:45.331 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/.snap/snap-subdir-test 2024-05-27T06:48:45.347 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/30 file after the snapshot 2024-05-27T06:48:45.349 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/31 file after the snapshot 2024-05-27T06:48:45.352 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/32 file after the snapshot 2024-05-27T06:48:45.786 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/33 file after the snapshot 2024-05-27T06:48:46.060 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/34 file after the snapshot 2024-05-27T06:48:46.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/35 file after the snapshot 2024-05-27T06:48:46.065 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/36 file after the snapshot 2024-05-27T06:48:46.067 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/37 file after the snapshot 2024-05-27T06:48:46.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/38 file after the snapshot 2024-05-27T06:48:46.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 149/39 file after the snapshot 2024-05-27T06:48:46.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/.snap/snap-subdir-test 2024-05-27T06:48:46.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/30 file after the snapshot 2024-05-27T06:48:46.117 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/31 file after the snapshot 2024-05-27T06:48:46.119 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/32 file after the snapshot 2024-05-27T06:48:46.121 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/33 file after the snapshot 2024-05-27T06:48:46.124 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/34 file after the snapshot 2024-05-27T06:48:46.126 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/35 file after the snapshot 2024-05-27T06:48:46.128 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/36 file after the snapshot 2024-05-27T06:48:46.131 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/37 file after the snapshot 2024-05-27T06:48:46.133 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/38 file after the snapshot 2024-05-27T06:48:46.135 INFO:tasks.workunit.client.0.smithi081.stdout: create 150/39 file after the snapshot 2024-05-27T06:48:46.138 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/.snap/snap-subdir-test 2024-05-27T06:48:46.229 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/30 file after the snapshot 2024-05-27T06:48:46.231 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/31 file after the snapshot 2024-05-27T06:48:46.233 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/32 file after the snapshot 2024-05-27T06:48:46.236 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/33 file after the snapshot 2024-05-27T06:48:46.238 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/34 file after the snapshot 2024-05-27T06:48:46.240 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/35 file after the snapshot 2024-05-27T06:48:46.242 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/36 file after the snapshot 2024-05-27T06:48:46.244 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/37 file after the snapshot 2024-05-27T06:48:46.247 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/38 file after the snapshot 2024-05-27T06:48:46.249 INFO:tasks.workunit.client.0.smithi081.stdout: create 151/39 file after the snapshot 2024-05-27T06:48:46.251 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/.snap/snap-subdir-test 2024-05-27T06:48:46.268 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/30 file after the snapshot 2024-05-27T06:48:46.270 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/31 file after the snapshot 2024-05-27T06:48:46.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/32 file after the snapshot 2024-05-27T06:48:46.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/33 file after the snapshot 2024-05-27T06:48:46.278 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/34 file after the snapshot 2024-05-27T06:48:46.280 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/35 file after the snapshot 2024-05-27T06:48:46.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/36 file after the snapshot 2024-05-27T06:48:46.285 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/37 file after the snapshot 2024-05-27T06:48:46.287 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/38 file after the snapshot 2024-05-27T06:48:46.289 INFO:tasks.workunit.client.0.smithi081.stdout: create 152/39 file after the snapshot 2024-05-27T06:48:46.291 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/.snap/snap-subdir-test 2024-05-27T06:48:46.308 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/30 file after the snapshot 2024-05-27T06:48:46.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/31 file after the snapshot 2024-05-27T06:48:46.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/32 file after the snapshot 2024-05-27T06:48:46.315 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/33 file after the snapshot 2024-05-27T06:48:46.317 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/34 file after the snapshot 2024-05-27T06:48:46.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/35 file after the snapshot 2024-05-27T06:48:46.323 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/36 file after the snapshot 2024-05-27T06:48:46.325 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/37 file after the snapshot 2024-05-27T06:48:46.328 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/38 file after the snapshot 2024-05-27T06:48:46.330 INFO:tasks.workunit.client.0.smithi081.stdout: create 153/39 file after the snapshot 2024-05-27T06:48:46.332 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/.snap/snap-subdir-test 2024-05-27T06:48:46.508 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/30 file after the snapshot 2024-05-27T06:48:46.510 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/31 file after the snapshot 2024-05-27T06:48:46.513 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/32 file after the snapshot 2024-05-27T06:48:46.515 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/33 file after the snapshot 2024-05-27T06:48:46.518 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/34 file after the snapshot 2024-05-27T06:48:46.521 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/35 file after the snapshot 2024-05-27T06:48:46.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/36 file after the snapshot 2024-05-27T06:48:46.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/37 file after the snapshot 2024-05-27T06:48:46.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/38 file after the snapshot 2024-05-27T06:48:46.530 INFO:tasks.workunit.client.0.smithi081.stdout: create 154/39 file after the snapshot 2024-05-27T06:48:46.533 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/.snap/snap-subdir-test 2024-05-27T06:48:47.290 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/30 file after the snapshot 2024-05-27T06:48:47.292 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/31 file after the snapshot 2024-05-27T06:48:47.295 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/32 file after the snapshot 2024-05-27T06:48:47.297 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/33 file after the snapshot 2024-05-27T06:48:47.300 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/34 file after the snapshot 2024-05-27T06:48:47.302 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/35 file after the snapshot 2024-05-27T06:48:47.305 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/36 file after the snapshot 2024-05-27T06:48:47.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/37 file after the snapshot 2024-05-27T06:48:47.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/38 file after the snapshot 2024-05-27T06:48:47.314 INFO:tasks.workunit.client.0.smithi081.stdout: create 155/39 file after the snapshot 2024-05-27T06:48:47.316 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/.snap/snap-subdir-test 2024-05-27T06:48:47.333 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/30 file after the snapshot 2024-05-27T06:48:47.336 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/31 file after the snapshot 2024-05-27T06:48:47.338 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/32 file after the snapshot 2024-05-27T06:48:47.340 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/33 file after the snapshot 2024-05-27T06:48:47.342 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/34 file after the snapshot 2024-05-27T06:48:47.345 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/35 file after the snapshot 2024-05-27T06:48:47.347 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/36 file after the snapshot 2024-05-27T06:48:47.349 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/37 file after the snapshot 2024-05-27T06:48:47.351 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/38 file after the snapshot 2024-05-27T06:48:47.353 INFO:tasks.workunit.client.0.smithi081.stdout: create 156/39 file after the snapshot 2024-05-27T06:48:47.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/.snap/snap-subdir-test 2024-05-27T06:48:47.371 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/30 file after the snapshot 2024-05-27T06:48:47.373 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/31 file after the snapshot 2024-05-27T06:48:47.376 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/32 file after the snapshot 2024-05-27T06:48:47.378 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/33 file after the snapshot 2024-05-27T06:48:47.380 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/34 file after the snapshot 2024-05-27T06:48:47.382 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/35 file after the snapshot 2024-05-27T06:48:47.384 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/36 file after the snapshot 2024-05-27T06:48:47.387 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/37 file after the snapshot 2024-05-27T06:48:47.389 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/38 file after the snapshot 2024-05-27T06:48:47.391 INFO:tasks.workunit.client.0.smithi081.stdout: create 157/39 file after the snapshot 2024-05-27T06:48:47.394 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/.snap/snap-subdir-test 2024-05-27T06:48:47.410 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/30 file after the snapshot 2024-05-27T06:48:47.412 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/31 file after the snapshot 2024-05-27T06:48:47.414 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/32 file after the snapshot 2024-05-27T06:48:47.416 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/33 file after the snapshot 2024-05-27T06:48:47.419 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/34 file after the snapshot 2024-05-27T06:48:47.421 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/35 file after the snapshot 2024-05-27T06:48:47.424 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/36 file after the snapshot 2024-05-27T06:48:47.426 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/37 file after the snapshot 2024-05-27T06:48:47.428 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/38 file after the snapshot 2024-05-27T06:48:47.430 INFO:tasks.workunit.client.0.smithi081.stdout: create 158/39 file after the snapshot 2024-05-27T06:48:47.433 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/.snap/snap-subdir-test 2024-05-27T06:48:47.450 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/30 file after the snapshot 2024-05-27T06:48:47.452 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/31 file after the snapshot 2024-05-27T06:48:47.455 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/32 file after the snapshot 2024-05-27T06:48:47.457 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/33 file after the snapshot 2024-05-27T06:48:47.459 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/34 file after the snapshot 2024-05-27T06:48:47.461 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/35 file after the snapshot 2024-05-27T06:48:47.464 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/36 file after the snapshot 2024-05-27T06:48:47.466 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/37 file after the snapshot 2024-05-27T06:48:47.468 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/38 file after the snapshot 2024-05-27T06:48:47.470 INFO:tasks.workunit.client.0.smithi081.stdout: create 159/39 file after the snapshot 2024-05-27T06:48:47.472 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/.snap/snap-subdir-test 2024-05-27T06:48:47.491 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/30 file after the snapshot 2024-05-27T06:48:47.494 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/31 file after the snapshot 2024-05-27T06:48:47.496 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/32 file after the snapshot 2024-05-27T06:48:47.498 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/33 file after the snapshot 2024-05-27T06:48:47.500 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/34 file after the snapshot 2024-05-27T06:48:47.503 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/35 file after the snapshot 2024-05-27T06:48:47.506 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/36 file after the snapshot 2024-05-27T06:48:47.508 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/37 file after the snapshot 2024-05-27T06:48:47.510 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/38 file after the snapshot 2024-05-27T06:48:47.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 160/39 file after the snapshot 2024-05-27T06:48:47.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/.snap/snap-subdir-test 2024-05-27T06:48:47.531 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/30 file after the snapshot 2024-05-27T06:48:47.533 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/31 file after the snapshot 2024-05-27T06:48:47.535 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/32 file after the snapshot 2024-05-27T06:48:47.538 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/33 file after the snapshot 2024-05-27T06:48:47.540 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/34 file after the snapshot 2024-05-27T06:48:47.542 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/35 file after the snapshot 2024-05-27T06:48:47.544 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/36 file after the snapshot 2024-05-27T06:48:47.546 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/37 file after the snapshot 2024-05-27T06:48:47.548 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/38 file after the snapshot 2024-05-27T06:48:47.551 INFO:tasks.workunit.client.0.smithi081.stdout: create 161/39 file after the snapshot 2024-05-27T06:48:47.553 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/.snap/snap-subdir-test 2024-05-27T06:48:47.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/30 file after the snapshot 2024-05-27T06:48:47.573 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/31 file after the snapshot 2024-05-27T06:48:47.576 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/32 file after the snapshot 2024-05-27T06:48:47.578 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/33 file after the snapshot 2024-05-27T06:48:47.580 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/34 file after the snapshot 2024-05-27T06:48:47.582 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/35 file after the snapshot 2024-05-27T06:48:47.585 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/36 file after the snapshot 2024-05-27T06:48:47.587 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/37 file after the snapshot 2024-05-27T06:48:47.590 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/38 file after the snapshot 2024-05-27T06:48:47.593 INFO:tasks.workunit.client.0.smithi081.stdout: create 162/39 file after the snapshot 2024-05-27T06:48:47.595 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/.snap/snap-subdir-test 2024-05-27T06:48:47.611 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/30 file after the snapshot 2024-05-27T06:48:47.613 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/31 file after the snapshot 2024-05-27T06:48:47.615 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/32 file after the snapshot 2024-05-27T06:48:47.618 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/33 file after the snapshot 2024-05-27T06:48:47.621 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/34 file after the snapshot 2024-05-27T06:48:47.624 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/35 file after the snapshot 2024-05-27T06:48:47.626 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/36 file after the snapshot 2024-05-27T06:48:47.628 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/37 file after the snapshot 2024-05-27T06:48:47.630 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/38 file after the snapshot 2024-05-27T06:48:47.632 INFO:tasks.workunit.client.0.smithi081.stdout: create 163/39 file after the snapshot 2024-05-27T06:48:47.634 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/.snap/snap-subdir-test 2024-05-27T06:48:47.650 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/30 file after the snapshot 2024-05-27T06:48:47.652 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/31 file after the snapshot 2024-05-27T06:48:47.655 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/32 file after the snapshot 2024-05-27T06:48:47.657 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/33 file after the snapshot 2024-05-27T06:48:47.660 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/34 file after the snapshot 2024-05-27T06:48:47.662 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/35 file after the snapshot 2024-05-27T06:48:47.665 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/36 file after the snapshot 2024-05-27T06:48:47.668 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/37 file after the snapshot 2024-05-27T06:48:47.670 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/38 file after the snapshot 2024-05-27T06:48:47.673 INFO:tasks.workunit.client.0.smithi081.stdout: create 164/39 file after the snapshot 2024-05-27T06:48:47.676 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/.snap/snap-subdir-test 2024-05-27T06:48:47.694 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/30 file after the snapshot 2024-05-27T06:48:47.697 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/31 file after the snapshot 2024-05-27T06:48:47.700 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/32 file after the snapshot 2024-05-27T06:48:47.702 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/33 file after the snapshot 2024-05-27T06:48:47.705 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/34 file after the snapshot 2024-05-27T06:48:47.707 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/35 file after the snapshot 2024-05-27T06:48:47.709 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/36 file after the snapshot 2024-05-27T06:48:47.712 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/37 file after the snapshot 2024-05-27T06:48:47.714 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/38 file after the snapshot 2024-05-27T06:48:47.717 INFO:tasks.workunit.client.0.smithi081.stdout: create 165/39 file after the snapshot 2024-05-27T06:48:47.719 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/.snap/snap-subdir-test 2024-05-27T06:48:47.738 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/30 file after the snapshot 2024-05-27T06:48:47.740 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/31 file after the snapshot 2024-05-27T06:48:47.742 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/32 file after the snapshot 2024-05-27T06:48:47.745 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/33 file after the snapshot 2024-05-27T06:48:47.747 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/34 file after the snapshot 2024-05-27T06:48:47.750 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/35 file after the snapshot 2024-05-27T06:48:47.752 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/36 file after the snapshot 2024-05-27T06:48:47.755 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/37 file after the snapshot 2024-05-27T06:48:47.758 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/38 file after the snapshot 2024-05-27T06:48:47.760 INFO:tasks.workunit.client.0.smithi081.stdout: create 166/39 file after the snapshot 2024-05-27T06:48:47.762 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/.snap/snap-subdir-test 2024-05-27T06:48:47.779 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/30 file after the snapshot 2024-05-27T06:48:47.782 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/31 file after the snapshot 2024-05-27T06:48:47.784 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/32 file after the snapshot 2024-05-27T06:48:47.786 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/33 file after the snapshot 2024-05-27T06:48:47.789 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/34 file after the snapshot 2024-05-27T06:48:47.793 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/35 file after the snapshot 2024-05-27T06:48:47.814 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/36 file after the snapshot 2024-05-27T06:48:47.817 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/37 file after the snapshot 2024-05-27T06:48:47.819 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/38 file after the snapshot 2024-05-27T06:48:47.822 INFO:tasks.workunit.client.0.smithi081.stdout: create 167/39 file after the snapshot 2024-05-27T06:48:47.824 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/.snap/snap-subdir-test 2024-05-27T06:48:47.853 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/30 file after the snapshot 2024-05-27T06:48:47.855 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/31 file after the snapshot 2024-05-27T06:48:47.858 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/32 file after the snapshot 2024-05-27T06:48:47.860 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/33 file after the snapshot 2024-05-27T06:48:47.863 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/34 file after the snapshot 2024-05-27T06:48:47.865 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/35 file after the snapshot 2024-05-27T06:48:47.867 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/36 file after the snapshot 2024-05-27T06:48:47.870 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/37 file after the snapshot 2024-05-27T06:48:47.872 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/38 file after the snapshot 2024-05-27T06:48:47.874 INFO:tasks.workunit.client.0.smithi081.stdout: create 168/39 file after the snapshot 2024-05-27T06:48:47.877 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/.snap/snap-subdir-test 2024-05-27T06:48:47.895 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/30 file after the snapshot 2024-05-27T06:48:47.897 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/31 file after the snapshot 2024-05-27T06:48:47.899 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/32 file after the snapshot 2024-05-27T06:48:47.901 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/33 file after the snapshot 2024-05-27T06:48:47.903 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/34 file after the snapshot 2024-05-27T06:48:47.905 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/35 file after the snapshot 2024-05-27T06:48:47.908 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/36 file after the snapshot 2024-05-27T06:48:47.910 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/37 file after the snapshot 2024-05-27T06:48:47.912 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/38 file after the snapshot 2024-05-27T06:48:47.914 INFO:tasks.workunit.client.0.smithi081.stdout: create 169/39 file after the snapshot 2024-05-27T06:48:47.917 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/.snap/snap-subdir-test 2024-05-27T06:48:47.936 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/30 file after the snapshot 2024-05-27T06:48:47.938 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/31 file after the snapshot 2024-05-27T06:48:47.941 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/32 file after the snapshot 2024-05-27T06:48:47.943 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/33 file after the snapshot 2024-05-27T06:48:47.946 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/34 file after the snapshot 2024-05-27T06:48:47.948 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/35 file after the snapshot 2024-05-27T06:48:47.951 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/36 file after the snapshot 2024-05-27T06:48:47.953 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/37 file after the snapshot 2024-05-27T06:48:47.955 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/38 file after the snapshot 2024-05-27T06:48:47.957 INFO:tasks.workunit.client.0.smithi081.stdout: create 170/39 file after the snapshot 2024-05-27T06:48:47.960 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/.snap/snap-subdir-test 2024-05-27T06:48:47.977 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/30 file after the snapshot 2024-05-27T06:48:47.979 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/31 file after the snapshot 2024-05-27T06:48:47.981 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/32 file after the snapshot 2024-05-27T06:48:47.983 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/33 file after the snapshot 2024-05-27T06:48:47.986 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/34 file after the snapshot 2024-05-27T06:48:47.988 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/35 file after the snapshot 2024-05-27T06:48:47.990 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/36 file after the snapshot 2024-05-27T06:48:47.993 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/37 file after the snapshot 2024-05-27T06:48:47.995 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/38 file after the snapshot 2024-05-27T06:48:47.998 INFO:tasks.workunit.client.0.smithi081.stdout: create 171/39 file after the snapshot 2024-05-27T06:48:48.001 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/.snap/snap-subdir-test 2024-05-27T06:48:48.018 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/30 file after the snapshot 2024-05-27T06:48:48.019 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/31 file after the snapshot 2024-05-27T06:48:48.022 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/32 file after the snapshot 2024-05-27T06:48:48.024 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/33 file after the snapshot 2024-05-27T06:48:48.026 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/34 file after the snapshot 2024-05-27T06:48:48.028 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/35 file after the snapshot 2024-05-27T06:48:48.030 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/36 file after the snapshot 2024-05-27T06:48:48.033 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/37 file after the snapshot 2024-05-27T06:48:48.035 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/38 file after the snapshot 2024-05-27T06:48:48.037 INFO:tasks.workunit.client.0.smithi081.stdout: create 172/39 file after the snapshot 2024-05-27T06:48:48.039 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/.snap/snap-subdir-test 2024-05-27T06:48:48.060 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/30 file after the snapshot 2024-05-27T06:48:48.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/31 file after the snapshot 2024-05-27T06:48:48.064 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/32 file after the snapshot 2024-05-27T06:48:48.066 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/33 file after the snapshot 2024-05-27T06:48:48.069 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/34 file after the snapshot 2024-05-27T06:48:48.071 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/35 file after the snapshot 2024-05-27T06:48:48.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/36 file after the snapshot 2024-05-27T06:48:48.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/37 file after the snapshot 2024-05-27T06:48:48.078 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/38 file after the snapshot 2024-05-27T06:48:48.080 INFO:tasks.workunit.client.0.smithi081.stdout: create 173/39 file after the snapshot 2024-05-27T06:48:48.083 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/.snap/snap-subdir-test 2024-05-27T06:48:48.101 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/30 file after the snapshot 2024-05-27T06:48:48.104 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/31 file after the snapshot 2024-05-27T06:48:48.106 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/32 file after the snapshot 2024-05-27T06:48:48.109 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/33 file after the snapshot 2024-05-27T06:48:48.111 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/34 file after the snapshot 2024-05-27T06:48:48.113 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/35 file after the snapshot 2024-05-27T06:48:48.115 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/36 file after the snapshot 2024-05-27T06:48:48.117 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/37 file after the snapshot 2024-05-27T06:48:48.119 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/38 file after the snapshot 2024-05-27T06:48:48.122 INFO:tasks.workunit.client.0.smithi081.stdout: create 174/39 file after the snapshot 2024-05-27T06:48:48.124 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/.snap/snap-subdir-test 2024-05-27T06:48:48.751 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/30 file after the snapshot 2024-05-27T06:48:48.754 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/31 file after the snapshot 2024-05-27T06:48:48.756 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/32 file after the snapshot 2024-05-27T06:48:48.759 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/33 file after the snapshot 2024-05-27T06:48:48.761 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/34 file after the snapshot 2024-05-27T06:48:48.764 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/35 file after the snapshot 2024-05-27T06:48:48.767 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/36 file after the snapshot 2024-05-27T06:48:48.769 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/37 file after the snapshot 2024-05-27T06:48:48.771 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/38 file after the snapshot 2024-05-27T06:48:48.774 INFO:tasks.workunit.client.0.smithi081.stdout: create 175/39 file after the snapshot 2024-05-27T06:48:48.776 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/.snap/snap-subdir-test 2024-05-27T06:48:48.794 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/30 file after the snapshot 2024-05-27T06:48:48.796 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/31 file after the snapshot 2024-05-27T06:48:48.799 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/32 file after the snapshot 2024-05-27T06:48:48.801 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/33 file after the snapshot 2024-05-27T06:48:48.803 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/34 file after the snapshot 2024-05-27T06:48:48.806 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/35 file after the snapshot 2024-05-27T06:48:48.808 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/36 file after the snapshot 2024-05-27T06:48:48.810 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/37 file after the snapshot 2024-05-27T06:48:48.813 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/38 file after the snapshot 2024-05-27T06:48:48.815 INFO:tasks.workunit.client.0.smithi081.stdout: create 176/39 file after the snapshot 2024-05-27T06:48:48.817 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/.snap/snap-subdir-test 2024-05-27T06:48:48.837 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/30 file after the snapshot 2024-05-27T06:48:48.839 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/31 file after the snapshot 2024-05-27T06:48:48.842 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/32 file after the snapshot 2024-05-27T06:48:48.844 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/33 file after the snapshot 2024-05-27T06:48:48.846 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/34 file after the snapshot 2024-05-27T06:48:48.849 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/35 file after the snapshot 2024-05-27T06:48:48.851 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/36 file after the snapshot 2024-05-27T06:48:48.853 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/37 file after the snapshot 2024-05-27T06:48:48.856 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/38 file after the snapshot 2024-05-27T06:48:48.859 INFO:tasks.workunit.client.0.smithi081.stdout: create 177/39 file after the snapshot 2024-05-27T06:48:48.861 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/.snap/snap-subdir-test 2024-05-27T06:48:48.878 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/30 file after the snapshot 2024-05-27T06:48:48.880 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/31 file after the snapshot 2024-05-27T06:48:48.882 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/32 file after the snapshot 2024-05-27T06:48:48.885 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/33 file after the snapshot 2024-05-27T06:48:48.887 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/34 file after the snapshot 2024-05-27T06:48:48.889 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/35 file after the snapshot 2024-05-27T06:48:48.891 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/36 file after the snapshot 2024-05-27T06:48:48.893 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/37 file after the snapshot 2024-05-27T06:48:48.895 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/38 file after the snapshot 2024-05-27T06:48:48.897 INFO:tasks.workunit.client.0.smithi081.stdout: create 178/39 file after the snapshot 2024-05-27T06:48:48.900 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/.snap/snap-subdir-test 2024-05-27T06:48:48.916 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/30 file after the snapshot 2024-05-27T06:48:48.918 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/31 file after the snapshot 2024-05-27T06:48:48.920 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/32 file after the snapshot 2024-05-27T06:48:48.922 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/33 file after the snapshot 2024-05-27T06:48:48.924 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/34 file after the snapshot 2024-05-27T06:48:48.926 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/35 file after the snapshot 2024-05-27T06:48:48.929 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/36 file after the snapshot 2024-05-27T06:48:48.931 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/37 file after the snapshot 2024-05-27T06:48:48.933 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/38 file after the snapshot 2024-05-27T06:48:48.935 INFO:tasks.workunit.client.0.smithi081.stdout: create 179/39 file after the snapshot 2024-05-27T06:48:48.937 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/.snap/snap-subdir-test 2024-05-27T06:48:48.954 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/30 file after the snapshot 2024-05-27T06:48:48.957 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/31 file after the snapshot 2024-05-27T06:48:48.960 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/32 file after the snapshot 2024-05-27T06:48:48.962 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/33 file after the snapshot 2024-05-27T06:48:48.965 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/34 file after the snapshot 2024-05-27T06:48:48.967 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/35 file after the snapshot 2024-05-27T06:48:48.970 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/36 file after the snapshot 2024-05-27T06:48:49.797 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/37 file after the snapshot 2024-05-27T06:48:49.818 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/38 file after the snapshot 2024-05-27T06:48:49.821 INFO:tasks.workunit.client.0.smithi081.stdout: create 180/39 file after the snapshot 2024-05-27T06:48:49.824 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/.snap/snap-subdir-test 2024-05-27T06:48:49.854 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/30 file after the snapshot 2024-05-27T06:48:49.856 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/31 file after the snapshot 2024-05-27T06:48:49.859 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/32 file after the snapshot 2024-05-27T06:48:49.861 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/33 file after the snapshot 2024-05-27T06:48:49.864 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/34 file after the snapshot 2024-05-27T06:48:49.866 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/35 file after the snapshot 2024-05-27T06:48:49.868 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/36 file after the snapshot 2024-05-27T06:48:49.871 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/37 file after the snapshot 2024-05-27T06:48:49.874 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/38 file after the snapshot 2024-05-27T06:48:50.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 181/39 file after the snapshot 2024-05-27T06:48:50.439 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/.snap/snap-subdir-test 2024-05-27T06:48:51.036 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/30 file after the snapshot 2024-05-27T06:48:51.038 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/31 file after the snapshot 2024-05-27T06:48:51.041 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/32 file after the snapshot 2024-05-27T06:48:51.044 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/33 file after the snapshot 2024-05-27T06:48:51.046 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/34 file after the snapshot 2024-05-27T06:48:51.049 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/35 file after the snapshot 2024-05-27T06:48:51.051 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/36 file after the snapshot 2024-05-27T06:48:51.054 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/37 file after the snapshot 2024-05-27T06:48:51.056 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/38 file after the snapshot 2024-05-27T06:48:51.059 INFO:tasks.workunit.client.0.smithi081.stdout: create 182/39 file after the snapshot 2024-05-27T06:48:51.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/.snap/snap-subdir-test 2024-05-27T06:48:51.081 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/30 file after the snapshot 2024-05-27T06:48:51.083 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/31 file after the snapshot 2024-05-27T06:48:51.085 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/32 file after the snapshot 2024-05-27T06:48:51.087 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/33 file after the snapshot 2024-05-27T06:48:51.090 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/34 file after the snapshot 2024-05-27T06:48:51.092 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/35 file after the snapshot 2024-05-27T06:48:51.155 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/36 file after the snapshot 2024-05-27T06:48:51.158 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/37 file after the snapshot 2024-05-27T06:48:51.160 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/38 file after the snapshot 2024-05-27T06:48:51.162 INFO:tasks.workunit.client.0.smithi081.stdout: create 183/39 file after the snapshot 2024-05-27T06:48:51.165 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/.snap/snap-subdir-test 2024-05-27T06:48:51.184 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/30 file after the snapshot 2024-05-27T06:48:51.186 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/31 file after the snapshot 2024-05-27T06:48:51.188 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/32 file after the snapshot 2024-05-27T06:48:51.191 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/33 file after the snapshot 2024-05-27T06:48:51.193 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/34 file after the snapshot 2024-05-27T06:48:51.195 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/35 file after the snapshot 2024-05-27T06:48:51.198 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/36 file after the snapshot 2024-05-27T06:48:51.200 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/37 file after the snapshot 2024-05-27T06:48:51.203 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/38 file after the snapshot 2024-05-27T06:48:51.205 INFO:tasks.workunit.client.0.smithi081.stdout: create 184/39 file after the snapshot 2024-05-27T06:48:51.207 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/.snap/snap-subdir-test 2024-05-27T06:48:51.225 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/30 file after the snapshot 2024-05-27T06:48:51.227 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/31 file after the snapshot 2024-05-27T06:48:51.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/32 file after the snapshot 2024-05-27T06:48:51.233 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/33 file after the snapshot 2024-05-27T06:48:51.236 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/34 file after the snapshot 2024-05-27T06:48:51.238 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/35 file after the snapshot 2024-05-27T06:48:51.240 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/36 file after the snapshot 2024-05-27T06:48:51.242 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/37 file after the snapshot 2024-05-27T06:48:51.245 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/38 file after the snapshot 2024-05-27T06:48:51.247 INFO:tasks.workunit.client.0.smithi081.stdout: create 185/39 file after the snapshot 2024-05-27T06:48:51.249 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/.snap/snap-subdir-test 2024-05-27T06:48:51.266 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/30 file after the snapshot 2024-05-27T06:48:51.268 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/31 file after the snapshot 2024-05-27T06:48:51.271 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/32 file after the snapshot 2024-05-27T06:48:51.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/33 file after the snapshot 2024-05-27T06:48:51.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/34 file after the snapshot 2024-05-27T06:48:51.277 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/35 file after the snapshot 2024-05-27T06:48:51.280 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/36 file after the snapshot 2024-05-27T06:48:51.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/37 file after the snapshot 2024-05-27T06:48:51.285 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/38 file after the snapshot 2024-05-27T06:48:51.287 INFO:tasks.workunit.client.0.smithi081.stdout: create 186/39 file after the snapshot 2024-05-27T06:48:51.290 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/.snap/snap-subdir-test 2024-05-27T06:48:51.309 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/30 file after the snapshot 2024-05-27T06:48:51.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/31 file after the snapshot 2024-05-27T06:48:51.315 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/32 file after the snapshot 2024-05-27T06:48:51.318 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/33 file after the snapshot 2024-05-27T06:48:51.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/34 file after the snapshot 2024-05-27T06:48:51.323 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/35 file after the snapshot 2024-05-27T06:48:51.326 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/36 file after the snapshot 2024-05-27T06:48:51.328 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/37 file after the snapshot 2024-05-27T06:48:51.331 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/38 file after the snapshot 2024-05-27T06:48:51.333 INFO:tasks.workunit.client.0.smithi081.stdout: create 187/39 file after the snapshot 2024-05-27T06:48:51.335 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/.snap/snap-subdir-test 2024-05-27T06:48:51.352 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/30 file after the snapshot 2024-05-27T06:48:51.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/31 file after the snapshot 2024-05-27T06:48:51.357 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/32 file after the snapshot 2024-05-27T06:48:51.359 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/33 file after the snapshot 2024-05-27T06:48:51.361 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/34 file after the snapshot 2024-05-27T06:48:51.364 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/35 file after the snapshot 2024-05-27T06:48:51.366 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/36 file after the snapshot 2024-05-27T06:48:51.368 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/37 file after the snapshot 2024-05-27T06:48:51.370 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/38 file after the snapshot 2024-05-27T06:48:51.373 INFO:tasks.workunit.client.0.smithi081.stdout: create 188/39 file after the snapshot 2024-05-27T06:48:51.375 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/.snap/snap-subdir-test 2024-05-27T06:48:51.392 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/30 file after the snapshot 2024-05-27T06:48:51.395 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/31 file after the snapshot 2024-05-27T06:48:51.397 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/32 file after the snapshot 2024-05-27T06:48:51.399 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/33 file after the snapshot 2024-05-27T06:48:51.402 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/34 file after the snapshot 2024-05-27T06:48:51.404 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/35 file after the snapshot 2024-05-27T06:48:51.406 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/36 file after the snapshot 2024-05-27T06:48:51.409 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/37 file after the snapshot 2024-05-27T06:48:51.411 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/38 file after the snapshot 2024-05-27T06:48:51.413 INFO:tasks.workunit.client.0.smithi081.stdout: create 189/39 file after the snapshot 2024-05-27T06:48:51.415 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/.snap/snap-subdir-test 2024-05-27T06:48:51.432 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/30 file after the snapshot 2024-05-27T06:48:51.434 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/31 file after the snapshot 2024-05-27T06:48:51.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/32 file after the snapshot 2024-05-27T06:48:51.439 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/33 file after the snapshot 2024-05-27T06:48:51.441 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/34 file after the snapshot 2024-05-27T06:48:51.444 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/35 file after the snapshot 2024-05-27T06:48:51.447 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/36 file after the snapshot 2024-05-27T06:48:51.449 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/37 file after the snapshot 2024-05-27T06:48:51.451 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/38 file after the snapshot 2024-05-27T06:48:51.453 INFO:tasks.workunit.client.0.smithi081.stdout: create 190/39 file after the snapshot 2024-05-27T06:48:51.456 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/.snap/snap-subdir-test 2024-05-27T06:48:51.474 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/30 file after the snapshot 2024-05-27T06:48:51.478 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/31 file after the snapshot 2024-05-27T06:48:51.478 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/32 file after the snapshot 2024-05-27T06:48:51.481 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/33 file after the snapshot 2024-05-27T06:48:51.483 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/34 file after the snapshot 2024-05-27T06:48:51.485 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/35 file after the snapshot 2024-05-27T06:48:51.488 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/36 file after the snapshot 2024-05-27T06:48:51.490 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/37 file after the snapshot 2024-05-27T06:48:51.492 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/38 file after the snapshot 2024-05-27T06:48:51.495 INFO:tasks.workunit.client.0.smithi081.stdout: create 191/39 file after the snapshot 2024-05-27T06:48:51.497 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/.snap/snap-subdir-test 2024-05-27T06:48:51.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/30 file after the snapshot 2024-05-27T06:48:51.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/31 file after the snapshot 2024-05-27T06:48:51.521 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/32 file after the snapshot 2024-05-27T06:48:51.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/33 file after the snapshot 2024-05-27T06:48:51.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/34 file after the snapshot 2024-05-27T06:48:51.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/35 file after the snapshot 2024-05-27T06:48:51.530 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/36 file after the snapshot 2024-05-27T06:48:51.532 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/37 file after the snapshot 2024-05-27T06:48:51.535 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/38 file after the snapshot 2024-05-27T06:48:51.537 INFO:tasks.workunit.client.0.smithi081.stdout: create 192/39 file after the snapshot 2024-05-27T06:48:51.539 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/.snap/snap-subdir-test 2024-05-27T06:48:51.557 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/30 file after the snapshot 2024-05-27T06:48:51.560 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/31 file after the snapshot 2024-05-27T06:48:51.562 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/32 file after the snapshot 2024-05-27T06:48:51.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/33 file after the snapshot 2024-05-27T06:48:51.567 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/34 file after the snapshot 2024-05-27T06:48:51.569 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/35 file after the snapshot 2024-05-27T06:48:51.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/36 file after the snapshot 2024-05-27T06:48:51.574 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/37 file after the snapshot 2024-05-27T06:48:51.576 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/38 file after the snapshot 2024-05-27T06:48:51.578 INFO:tasks.workunit.client.0.smithi081.stdout: create 193/39 file after the snapshot 2024-05-27T06:48:51.580 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/.snap/snap-subdir-test 2024-05-27T06:48:51.598 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/30 file after the snapshot 2024-05-27T06:48:51.600 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/31 file after the snapshot 2024-05-27T06:48:51.804 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/32 file after the snapshot 2024-05-27T06:48:52.063 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/33 file after the snapshot 2024-05-27T06:48:52.065 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/34 file after the snapshot 2024-05-27T06:48:52.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/35 file after the snapshot 2024-05-27T06:48:52.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/36 file after the snapshot 2024-05-27T06:48:52.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/37 file after the snapshot 2024-05-27T06:48:52.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/38 file after the snapshot 2024-05-27T06:48:52.078 INFO:tasks.workunit.client.0.smithi081.stdout: create 194/39 file after the snapshot 2024-05-27T06:48:52.080 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/.snap/snap-subdir-test 2024-05-27T06:48:52.102 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/30 file after the snapshot 2024-05-27T06:48:52.104 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/31 file after the snapshot 2024-05-27T06:48:52.106 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/32 file after the snapshot 2024-05-27T06:48:52.109 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/33 file after the snapshot 2024-05-27T06:48:52.111 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/34 file after the snapshot 2024-05-27T06:48:52.113 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/35 file after the snapshot 2024-05-27T06:48:52.115 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/36 file after the snapshot 2024-05-27T06:48:52.118 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/37 file after the snapshot 2024-05-27T06:48:52.120 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/38 file after the snapshot 2024-05-27T06:48:52.122 INFO:tasks.workunit.client.0.smithi081.stdout: create 195/39 file after the snapshot 2024-05-27T06:48:52.124 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/.snap/snap-subdir-test 2024-05-27T06:48:52.141 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/30 file after the snapshot 2024-05-27T06:48:52.143 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/31 file after the snapshot 2024-05-27T06:48:52.808 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/32 file after the snapshot 2024-05-27T06:48:53.080 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/33 file after the snapshot 2024-05-27T06:48:53.082 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/34 file after the snapshot 2024-05-27T06:48:53.085 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/35 file after the snapshot 2024-05-27T06:48:53.088 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/36 file after the snapshot 2024-05-27T06:48:53.090 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/37 file after the snapshot 2024-05-27T06:48:53.093 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/38 file after the snapshot 2024-05-27T06:48:53.096 INFO:tasks.workunit.client.0.smithi081.stdout: create 196/39 file after the snapshot 2024-05-27T06:48:53.099 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/.snap/snap-subdir-test 2024-05-27T06:48:53.138 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/30 file after the snapshot 2024-05-27T06:48:53.140 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/31 file after the snapshot 2024-05-27T06:48:53.144 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/32 file after the snapshot 2024-05-27T06:48:53.146 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/33 file after the snapshot 2024-05-27T06:48:53.149 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/34 file after the snapshot 2024-05-27T06:48:53.151 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/35 file after the snapshot 2024-05-27T06:48:53.153 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/36 file after the snapshot 2024-05-27T06:48:53.156 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/37 file after the snapshot 2024-05-27T06:48:53.158 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/38 file after the snapshot 2024-05-27T06:48:53.160 INFO:tasks.workunit.client.0.smithi081.stdout: create 197/39 file after the snapshot 2024-05-27T06:48:53.162 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/.snap/snap-subdir-test 2024-05-27T06:48:53.182 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/30 file after the snapshot 2024-05-27T06:48:53.184 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/31 file after the snapshot 2024-05-27T06:48:53.186 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/32 file after the snapshot 2024-05-27T06:48:53.189 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/33 file after the snapshot 2024-05-27T06:48:53.192 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/34 file after the snapshot 2024-05-27T06:48:53.195 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/35 file after the snapshot 2024-05-27T06:48:53.198 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/36 file after the snapshot 2024-05-27T06:48:53.200 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/37 file after the snapshot 2024-05-27T06:48:53.203 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/38 file after the snapshot 2024-05-27T06:48:53.206 INFO:tasks.workunit.client.0.smithi081.stdout: create 198/39 file after the snapshot 2024-05-27T06:48:53.209 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/.snap/snap-subdir-test 2024-05-27T06:48:53.227 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/30 file after the snapshot 2024-05-27T06:48:53.229 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/31 file after the snapshot 2024-05-27T06:48:53.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/32 file after the snapshot 2024-05-27T06:48:53.234 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/33 file after the snapshot 2024-05-27T06:48:53.236 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/34 file after the snapshot 2024-05-27T06:48:53.239 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/35 file after the snapshot 2024-05-27T06:48:53.242 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/36 file after the snapshot 2024-05-27T06:48:53.244 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/37 file after the snapshot 2024-05-27T06:48:53.246 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/38 file after the snapshot 2024-05-27T06:48:53.249 INFO:tasks.workunit.client.0.smithi081.stdout: create 199/39 file after the snapshot 2024-05-27T06:48:53.251 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/.snap/snap-subdir-test 2024-05-27T06:48:53.268 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/30 file after the snapshot 2024-05-27T06:48:53.270 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/31 file after the snapshot 2024-05-27T06:48:53.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/32 file after the snapshot 2024-05-27T06:48:53.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/33 file after the snapshot 2024-05-27T06:48:53.277 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/34 file after the snapshot 2024-05-27T06:48:53.279 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/35 file after the snapshot 2024-05-27T06:48:53.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/36 file after the snapshot 2024-05-27T06:48:53.284 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/37 file after the snapshot 2024-05-27T06:48:53.287 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/38 file after the snapshot 2024-05-27T06:48:53.290 INFO:tasks.workunit.client.0.smithi081.stdout: create 200/39 file after the snapshot 2024-05-27T06:48:53.292 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/.snap/snap-subdir-test 2024-05-27T06:48:53.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/30 file after the snapshot 2024-05-27T06:48:53.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/31 file after the snapshot 2024-05-27T06:48:53.315 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/32 file after the snapshot 2024-05-27T06:48:53.317 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/33 file after the snapshot 2024-05-27T06:48:53.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/34 file after the snapshot 2024-05-27T06:48:53.322 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/35 file after the snapshot 2024-05-27T06:48:53.324 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/36 file after the snapshot 2024-05-27T06:48:53.326 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/37 file after the snapshot 2024-05-27T06:48:53.329 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/38 file after the snapshot 2024-05-27T06:48:53.331 INFO:tasks.workunit.client.0.smithi081.stdout: create 201/39 file after the snapshot 2024-05-27T06:48:53.334 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/.snap/snap-subdir-test 2024-05-27T06:48:53.351 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/30 file after the snapshot 2024-05-27T06:48:53.353 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/31 file after the snapshot 2024-05-27T06:48:53.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/32 file after the snapshot 2024-05-27T06:48:53.358 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/33 file after the snapshot 2024-05-27T06:48:53.360 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/34 file after the snapshot 2024-05-27T06:48:53.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/35 file after the snapshot 2024-05-27T06:48:53.364 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/36 file after the snapshot 2024-05-27T06:48:53.366 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/37 file after the snapshot 2024-05-27T06:48:53.369 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/38 file after the snapshot 2024-05-27T06:48:53.371 INFO:tasks.workunit.client.0.smithi081.stdout: create 202/39 file after the snapshot 2024-05-27T06:48:53.373 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/.snap/snap-subdir-test 2024-05-27T06:48:53.390 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/30 file after the snapshot 2024-05-27T06:48:53.392 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/31 file after the snapshot 2024-05-27T06:48:53.394 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/32 file after the snapshot 2024-05-27T06:48:53.397 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/33 file after the snapshot 2024-05-27T06:48:53.399 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/34 file after the snapshot 2024-05-27T06:48:53.401 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/35 file after the snapshot 2024-05-27T06:48:53.403 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/36 file after the snapshot 2024-05-27T06:48:53.406 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/37 file after the snapshot 2024-05-27T06:48:53.408 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/38 file after the snapshot 2024-05-27T06:48:53.410 INFO:tasks.workunit.client.0.smithi081.stdout: create 203/39 file after the snapshot 2024-05-27T06:48:53.413 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/.snap/snap-subdir-test 2024-05-27T06:48:53.429 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/30 file after the snapshot 2024-05-27T06:48:53.431 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/31 file after the snapshot 2024-05-27T06:48:53.434 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/32 file after the snapshot 2024-05-27T06:48:53.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/33 file after the snapshot 2024-05-27T06:48:53.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/34 file after the snapshot 2024-05-27T06:48:53.441 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/35 file after the snapshot 2024-05-27T06:48:53.443 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/36 file after the snapshot 2024-05-27T06:48:53.446 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/37 file after the snapshot 2024-05-27T06:48:53.448 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/38 file after the snapshot 2024-05-27T06:48:53.451 INFO:tasks.workunit.client.0.smithi081.stdout: create 204/39 file after the snapshot 2024-05-27T06:48:53.454 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/.snap/snap-subdir-test 2024-05-27T06:48:53.922 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/30 file after the snapshot 2024-05-27T06:48:53.924 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/31 file after the snapshot 2024-05-27T06:48:53.926 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/32 file after the snapshot 2024-05-27T06:48:53.928 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/33 file after the snapshot 2024-05-27T06:48:53.933 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/34 file after the snapshot 2024-05-27T06:48:53.933 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/35 file after the snapshot 2024-05-27T06:48:53.935 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/36 file after the snapshot 2024-05-27T06:48:53.937 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/37 file after the snapshot 2024-05-27T06:48:53.940 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/38 file after the snapshot 2024-05-27T06:48:53.942 INFO:tasks.workunit.client.0.smithi081.stdout: create 205/39 file after the snapshot 2024-05-27T06:48:53.945 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/.snap/snap-subdir-test 2024-05-27T06:48:54.072 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/30 file after the snapshot 2024-05-27T06:48:54.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/31 file after the snapshot 2024-05-27T06:48:54.077 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/32 file after the snapshot 2024-05-27T06:48:54.081 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/33 file after the snapshot 2024-05-27T06:48:54.084 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/34 file after the snapshot 2024-05-27T06:48:54.086 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/35 file after the snapshot 2024-05-27T06:48:54.089 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/36 file after the snapshot 2024-05-27T06:48:54.091 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/37 file after the snapshot 2024-05-27T06:48:54.093 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/38 file after the snapshot 2024-05-27T06:48:54.095 INFO:tasks.workunit.client.0.smithi081.stdout: create 206/39 file after the snapshot 2024-05-27T06:48:54.098 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/.snap/snap-subdir-test 2024-05-27T06:48:54.405 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/30 file after the snapshot 2024-05-27T06:48:54.407 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/31 file after the snapshot 2024-05-27T06:48:54.409 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/32 file after the snapshot 2024-05-27T06:48:54.412 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/33 file after the snapshot 2024-05-27T06:48:54.414 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/34 file after the snapshot 2024-05-27T06:48:54.417 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/35 file after the snapshot 2024-05-27T06:48:54.419 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/36 file after the snapshot 2024-05-27T06:48:54.422 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/37 file after the snapshot 2024-05-27T06:48:54.424 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/38 file after the snapshot 2024-05-27T06:48:54.427 INFO:tasks.workunit.client.0.smithi081.stdout: create 207/39 file after the snapshot 2024-05-27T06:48:54.429 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/.snap/snap-subdir-test 2024-05-27T06:48:54.446 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/30 file after the snapshot 2024-05-27T06:48:54.449 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/31 file after the snapshot 2024-05-27T06:48:54.451 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/32 file after the snapshot 2024-05-27T06:48:54.453 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/33 file after the snapshot 2024-05-27T06:48:54.456 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/34 file after the snapshot 2024-05-27T06:48:54.458 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/35 file after the snapshot 2024-05-27T06:48:54.460 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/36 file after the snapshot 2024-05-27T06:48:54.462 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/37 file after the snapshot 2024-05-27T06:48:54.465 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/38 file after the snapshot 2024-05-27T06:48:54.467 INFO:tasks.workunit.client.0.smithi081.stdout: create 208/39 file after the snapshot 2024-05-27T06:48:54.469 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/.snap/snap-subdir-test 2024-05-27T06:48:54.486 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/30 file after the snapshot 2024-05-27T06:48:54.488 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/31 file after the snapshot 2024-05-27T06:48:54.491 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/32 file after the snapshot 2024-05-27T06:48:54.493 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/33 file after the snapshot 2024-05-27T06:48:54.495 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/34 file after the snapshot 2024-05-27T06:48:54.497 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/35 file after the snapshot 2024-05-27T06:48:54.499 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/36 file after the snapshot 2024-05-27T06:48:54.502 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/37 file after the snapshot 2024-05-27T06:48:54.504 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/38 file after the snapshot 2024-05-27T06:48:54.506 INFO:tasks.workunit.client.0.smithi081.stdout: create 209/39 file after the snapshot 2024-05-27T06:48:54.508 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/.snap/snap-subdir-test 2024-05-27T06:48:54.525 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/30 file after the snapshot 2024-05-27T06:48:54.527 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/31 file after the snapshot 2024-05-27T06:48:54.530 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/32 file after the snapshot 2024-05-27T06:48:54.532 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/33 file after the snapshot 2024-05-27T06:48:54.535 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/34 file after the snapshot 2024-05-27T06:48:54.537 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/35 file after the snapshot 2024-05-27T06:48:54.539 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/36 file after the snapshot 2024-05-27T06:48:54.541 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/37 file after the snapshot 2024-05-27T06:48:54.544 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/38 file after the snapshot 2024-05-27T06:48:54.546 INFO:tasks.workunit.client.0.smithi081.stdout: create 210/39 file after the snapshot 2024-05-27T06:48:54.548 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/.snap/snap-subdir-test 2024-05-27T06:48:54.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/30 file after the snapshot 2024-05-27T06:48:54.567 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/31 file after the snapshot 2024-05-27T06:48:54.569 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/32 file after the snapshot 2024-05-27T06:48:54.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/33 file after the snapshot 2024-05-27T06:48:54.573 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/34 file after the snapshot 2024-05-27T06:48:54.575 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/35 file after the snapshot 2024-05-27T06:48:54.577 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/36 file after the snapshot 2024-05-27T06:48:54.579 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/37 file after the snapshot 2024-05-27T06:48:54.582 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/38 file after the snapshot 2024-05-27T06:48:54.584 INFO:tasks.workunit.client.0.smithi081.stdout: create 211/39 file after the snapshot 2024-05-27T06:48:54.586 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/.snap/snap-subdir-test 2024-05-27T06:48:54.603 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/30 file after the snapshot 2024-05-27T06:48:54.605 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/31 file after the snapshot 2024-05-27T06:48:54.607 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/32 file after the snapshot 2024-05-27T06:48:54.609 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/33 file after the snapshot 2024-05-27T06:48:54.611 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/34 file after the snapshot 2024-05-27T06:48:54.614 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/35 file after the snapshot 2024-05-27T06:48:54.616 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/36 file after the snapshot 2024-05-27T06:48:54.618 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/37 file after the snapshot 2024-05-27T06:48:54.620 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/38 file after the snapshot 2024-05-27T06:48:54.623 INFO:tasks.workunit.client.0.smithi081.stdout: create 212/39 file after the snapshot 2024-05-27T06:48:54.625 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/.snap/snap-subdir-test 2024-05-27T06:48:54.645 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/30 file after the snapshot 2024-05-27T06:48:54.648 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/31 file after the snapshot 2024-05-27T06:48:54.650 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/32 file after the snapshot 2024-05-27T06:48:54.652 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/33 file after the snapshot 2024-05-27T06:48:54.655 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/34 file after the snapshot 2024-05-27T06:48:54.657 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/35 file after the snapshot 2024-05-27T06:48:54.660 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/36 file after the snapshot 2024-05-27T06:48:54.662 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/37 file after the snapshot 2024-05-27T06:48:54.664 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/38 file after the snapshot 2024-05-27T06:48:54.666 INFO:tasks.workunit.client.0.smithi081.stdout: create 213/39 file after the snapshot 2024-05-27T06:48:54.668 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/.snap/snap-subdir-test 2024-05-27T06:48:54.685 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/30 file after the snapshot 2024-05-27T06:48:54.687 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/31 file after the snapshot 2024-05-27T06:48:54.689 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/32 file after the snapshot 2024-05-27T06:48:54.692 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/33 file after the snapshot 2024-05-27T06:48:54.694 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/34 file after the snapshot 2024-05-27T06:48:54.697 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/35 file after the snapshot 2024-05-27T06:48:54.699 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/36 file after the snapshot 2024-05-27T06:48:54.814 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/37 file after the snapshot 2024-05-27T06:48:55.023 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/38 file after the snapshot 2024-05-27T06:48:55.025 INFO:tasks.workunit.client.0.smithi081.stdout: create 214/39 file after the snapshot 2024-05-27T06:48:55.028 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/.snap/snap-subdir-test 2024-05-27T06:48:55.052 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/30 file after the snapshot 2024-05-27T06:48:55.054 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/31 file after the snapshot 2024-05-27T06:48:55.057 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/32 file after the snapshot 2024-05-27T06:48:55.059 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/33 file after the snapshot 2024-05-27T06:48:55.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/34 file after the snapshot 2024-05-27T06:48:55.064 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/35 file after the snapshot 2024-05-27T06:48:55.066 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/36 file after the snapshot 2024-05-27T06:48:55.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/37 file after the snapshot 2024-05-27T06:48:55.071 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/38 file after the snapshot 2024-05-27T06:48:55.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 215/39 file after the snapshot 2024-05-27T06:48:55.076 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/.snap/snap-subdir-test 2024-05-27T06:48:55.092 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/30 file after the snapshot 2024-05-27T06:48:55.094 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/31 file after the snapshot 2024-05-27T06:48:55.097 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/32 file after the snapshot 2024-05-27T06:48:55.099 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/33 file after the snapshot 2024-05-27T06:48:55.102 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/34 file after the snapshot 2024-05-27T06:48:55.104 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/35 file after the snapshot 2024-05-27T06:48:55.106 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/36 file after the snapshot 2024-05-27T06:48:55.108 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/37 file after the snapshot 2024-05-27T06:48:55.110 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/38 file after the snapshot 2024-05-27T06:48:55.113 INFO:tasks.workunit.client.0.smithi081.stdout: create 216/39 file after the snapshot 2024-05-27T06:48:55.115 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/.snap/snap-subdir-test 2024-05-27T06:48:55.569 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/30 file after the snapshot 2024-05-27T06:48:55.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/31 file after the snapshot 2024-05-27T06:48:55.574 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/32 file after the snapshot 2024-05-27T06:48:55.576 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/33 file after the snapshot 2024-05-27T06:48:55.578 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/34 file after the snapshot 2024-05-27T06:48:55.581 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/35 file after the snapshot 2024-05-27T06:48:55.583 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/36 file after the snapshot 2024-05-27T06:48:55.585 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/37 file after the snapshot 2024-05-27T06:48:55.588 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/38 file after the snapshot 2024-05-27T06:48:55.590 INFO:tasks.workunit.client.0.smithi081.stdout: create 217/39 file after the snapshot 2024-05-27T06:48:55.593 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/.snap/snap-subdir-test 2024-05-27T06:48:55.897 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/30 file after the snapshot 2024-05-27T06:48:55.900 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/31 file after the snapshot 2024-05-27T06:48:55.903 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/32 file after the snapshot 2024-05-27T06:48:55.905 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/33 file after the snapshot 2024-05-27T06:48:56.784 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/34 file after the snapshot 2024-05-27T06:48:56.786 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/35 file after the snapshot 2024-05-27T06:48:56.789 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/36 file after the snapshot 2024-05-27T06:48:56.792 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/37 file after the snapshot 2024-05-27T06:48:56.795 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/38 file after the snapshot 2024-05-27T06:48:56.798 INFO:tasks.workunit.client.0.smithi081.stdout: create 218/39 file after the snapshot 2024-05-27T06:48:56.802 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/.snap/snap-subdir-test 2024-05-27T06:48:56.823 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/30 file after the snapshot 2024-05-27T06:48:56.868 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/31 file after the snapshot 2024-05-27T06:48:56.870 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/32 file after the snapshot 2024-05-27T06:48:56.873 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/33 file after the snapshot 2024-05-27T06:48:56.876 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/34 file after the snapshot 2024-05-27T06:48:56.878 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/35 file after the snapshot 2024-05-27T06:48:56.881 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/36 file after the snapshot 2024-05-27T06:48:56.883 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/37 file after the snapshot 2024-05-27T06:48:56.886 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/38 file after the snapshot 2024-05-27T06:48:56.888 INFO:tasks.workunit.client.0.smithi081.stdout: create 219/39 file after the snapshot 2024-05-27T06:48:56.891 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/.snap/snap-subdir-test 2024-05-27T06:48:56.931 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/30 file after the snapshot 2024-05-27T06:48:56.933 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/31 file after the snapshot 2024-05-27T06:48:56.935 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/32 file after the snapshot 2024-05-27T06:48:56.938 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/33 file after the snapshot 2024-05-27T06:48:56.940 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/34 file after the snapshot 2024-05-27T06:48:56.943 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/35 file after the snapshot 2024-05-27T06:48:56.945 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/36 file after the snapshot 2024-05-27T06:48:56.948 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/37 file after the snapshot 2024-05-27T06:48:56.950 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/38 file after the snapshot 2024-05-27T06:48:56.952 INFO:tasks.workunit.client.0.smithi081.stdout: create 220/39 file after the snapshot 2024-05-27T06:48:56.954 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/.snap/snap-subdir-test 2024-05-27T06:48:56.971 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/30 file after the snapshot 2024-05-27T06:48:56.974 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/31 file after the snapshot 2024-05-27T06:48:56.976 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/32 file after the snapshot 2024-05-27T06:48:56.979 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/33 file after the snapshot 2024-05-27T06:48:56.981 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/34 file after the snapshot 2024-05-27T06:48:56.983 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/35 file after the snapshot 2024-05-27T06:48:56.986 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/36 file after the snapshot 2024-05-27T06:48:56.988 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/37 file after the snapshot 2024-05-27T06:48:56.990 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/38 file after the snapshot 2024-05-27T06:48:56.992 INFO:tasks.workunit.client.0.smithi081.stdout: create 221/39 file after the snapshot 2024-05-27T06:48:56.995 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/.snap/snap-subdir-test 2024-05-27T06:48:57.013 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/30 file after the snapshot 2024-05-27T06:48:57.015 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/31 file after the snapshot 2024-05-27T06:48:57.017 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/32 file after the snapshot 2024-05-27T06:48:57.019 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/33 file after the snapshot 2024-05-27T06:48:57.022 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/34 file after the snapshot 2024-05-27T06:48:57.024 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/35 file after the snapshot 2024-05-27T06:48:57.026 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/36 file after the snapshot 2024-05-27T06:48:57.028 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/37 file after the snapshot 2024-05-27T06:48:57.031 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/38 file after the snapshot 2024-05-27T06:48:57.033 INFO:tasks.workunit.client.0.smithi081.stdout: create 222/39 file after the snapshot 2024-05-27T06:48:57.035 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/.snap/snap-subdir-test 2024-05-27T06:48:57.053 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/30 file after the snapshot 2024-05-27T06:48:57.054 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/31 file after the snapshot 2024-05-27T06:48:57.057 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/32 file after the snapshot 2024-05-27T06:48:57.059 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/33 file after the snapshot 2024-05-27T06:48:57.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/34 file after the snapshot 2024-05-27T06:48:57.064 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/35 file after the snapshot 2024-05-27T06:48:57.067 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/36 file after the snapshot 2024-05-27T06:48:57.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/37 file after the snapshot 2024-05-27T06:48:57.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/38 file after the snapshot 2024-05-27T06:48:57.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 223/39 file after the snapshot 2024-05-27T06:48:57.078 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/.snap/snap-subdir-test 2024-05-27T06:48:57.125 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/30 file after the snapshot 2024-05-27T06:48:57.127 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/31 file after the snapshot 2024-05-27T06:48:57.129 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/32 file after the snapshot 2024-05-27T06:48:57.132 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/33 file after the snapshot 2024-05-27T06:48:57.134 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/34 file after the snapshot 2024-05-27T06:48:57.137 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/35 file after the snapshot 2024-05-27T06:48:57.139 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/36 file after the snapshot 2024-05-27T06:48:57.141 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/37 file after the snapshot 2024-05-27T06:48:57.143 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/38 file after the snapshot 2024-05-27T06:48:57.145 INFO:tasks.workunit.client.0.smithi081.stdout: create 224/39 file after the snapshot 2024-05-27T06:48:57.147 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/.snap/snap-subdir-test 2024-05-27T06:48:57.164 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/30 file after the snapshot 2024-05-27T06:48:57.821 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/31 file after the snapshot 2024-05-27T06:48:58.063 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/32 file after the snapshot 2024-05-27T06:48:58.065 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/33 file after the snapshot 2024-05-27T06:48:58.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/34 file after the snapshot 2024-05-27T06:48:58.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/35 file after the snapshot 2024-05-27T06:48:58.072 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/36 file after the snapshot 2024-05-27T06:48:58.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/37 file after the snapshot 2024-05-27T06:48:58.077 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/38 file after the snapshot 2024-05-27T06:48:58.079 INFO:tasks.workunit.client.0.smithi081.stdout: create 225/39 file after the snapshot 2024-05-27T06:48:58.082 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/.snap/snap-subdir-test 2024-05-27T06:48:58.103 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/30 file after the snapshot 2024-05-27T06:48:58.105 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/31 file after the snapshot 2024-05-27T06:48:58.107 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/32 file after the snapshot 2024-05-27T06:48:58.109 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/33 file after the snapshot 2024-05-27T06:48:58.111 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/34 file after the snapshot 2024-05-27T06:48:58.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/35 file after the snapshot 2024-05-27T06:48:58.116 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/36 file after the snapshot 2024-05-27T06:48:58.119 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/37 file after the snapshot 2024-05-27T06:48:58.121 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/38 file after the snapshot 2024-05-27T06:48:58.123 INFO:tasks.workunit.client.0.smithi081.stdout: create 226/39 file after the snapshot 2024-05-27T06:48:58.125 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/.snap/snap-subdir-test 2024-05-27T06:48:58.144 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/30 file after the snapshot 2024-05-27T06:48:58.146 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/31 file after the snapshot 2024-05-27T06:48:58.149 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/32 file after the snapshot 2024-05-27T06:48:58.151 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/33 file after the snapshot 2024-05-27T06:48:58.153 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/34 file after the snapshot 2024-05-27T06:48:58.156 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/35 file after the snapshot 2024-05-27T06:48:58.157 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/36 file after the snapshot 2024-05-27T06:48:58.160 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/37 file after the snapshot 2024-05-27T06:48:58.162 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/38 file after the snapshot 2024-05-27T06:48:58.164 INFO:tasks.workunit.client.0.smithi081.stdout: create 227/39 file after the snapshot 2024-05-27T06:48:58.167 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/.snap/snap-subdir-test 2024-05-27T06:48:58.185 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/30 file after the snapshot 2024-05-27T06:48:58.187 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/31 file after the snapshot 2024-05-27T06:48:58.191 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/32 file after the snapshot 2024-05-27T06:48:58.193 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/33 file after the snapshot 2024-05-27T06:48:58.195 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/34 file after the snapshot 2024-05-27T06:48:58.197 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/35 file after the snapshot 2024-05-27T06:48:58.199 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/36 file after the snapshot 2024-05-27T06:48:58.201 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/37 file after the snapshot 2024-05-27T06:48:58.204 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/38 file after the snapshot 2024-05-27T06:48:58.206 INFO:tasks.workunit.client.0.smithi081.stdout: create 228/39 file after the snapshot 2024-05-27T06:48:58.208 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/.snap/snap-subdir-test 2024-05-27T06:48:58.225 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/30 file after the snapshot 2024-05-27T06:48:58.227 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/31 file after the snapshot 2024-05-27T06:48:58.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/32 file after the snapshot 2024-05-27T06:48:58.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/33 file after the snapshot 2024-05-27T06:48:58.234 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/34 file after the snapshot 2024-05-27T06:48:58.236 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/35 file after the snapshot 2024-05-27T06:48:58.238 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/36 file after the snapshot 2024-05-27T06:48:58.242 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/37 file after the snapshot 2024-05-27T06:48:58.243 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/38 file after the snapshot 2024-05-27T06:48:58.245 INFO:tasks.workunit.client.0.smithi081.stdout: create 229/39 file after the snapshot 2024-05-27T06:48:58.248 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/.snap/snap-subdir-test 2024-05-27T06:48:58.507 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/30 file after the snapshot 2024-05-27T06:48:58.509 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/31 file after the snapshot 2024-05-27T06:48:58.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/32 file after the snapshot 2024-05-27T06:48:58.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/33 file after the snapshot 2024-05-27T06:48:58.516 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/34 file after the snapshot 2024-05-27T06:48:58.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/35 file after the snapshot 2024-05-27T06:48:58.521 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/36 file after the snapshot 2024-05-27T06:48:58.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/37 file after the snapshot 2024-05-27T06:48:58.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/38 file after the snapshot 2024-05-27T06:48:58.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 230/39 file after the snapshot 2024-05-27T06:48:58.530 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/.snap/snap-subdir-test 2024-05-27T06:48:58.548 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/30 file after the snapshot 2024-05-27T06:48:58.550 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/31 file after the snapshot 2024-05-27T06:48:58.553 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/32 file after the snapshot 2024-05-27T06:48:58.555 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/33 file after the snapshot 2024-05-27T06:48:58.557 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/34 file after the snapshot 2024-05-27T06:48:58.559 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/35 file after the snapshot 2024-05-27T06:48:58.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/36 file after the snapshot 2024-05-27T06:48:58.563 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/37 file after the snapshot 2024-05-27T06:48:58.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/38 file after the snapshot 2024-05-27T06:48:58.568 INFO:tasks.workunit.client.0.smithi081.stdout: create 231/39 file after the snapshot 2024-05-27T06:48:58.570 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/.snap/snap-subdir-test 2024-05-27T06:48:58.587 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/30 file after the snapshot 2024-05-27T06:48:58.589 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/31 file after the snapshot 2024-05-27T06:48:58.592 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/32 file after the snapshot 2024-05-27T06:48:58.594 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/33 file after the snapshot 2024-05-27T06:48:58.596 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/34 file after the snapshot 2024-05-27T06:48:58.598 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/35 file after the snapshot 2024-05-27T06:48:58.600 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/36 file after the snapshot 2024-05-27T06:48:58.603 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/37 file after the snapshot 2024-05-27T06:48:58.605 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/38 file after the snapshot 2024-05-27T06:48:58.607 INFO:tasks.workunit.client.0.smithi081.stdout: create 232/39 file after the snapshot 2024-05-27T06:48:58.610 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/.snap/snap-subdir-test 2024-05-27T06:48:59.107 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/30 file after the snapshot 2024-05-27T06:48:59.786 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/31 file after the snapshot 2024-05-27T06:48:59.788 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/32 file after the snapshot 2024-05-27T06:48:59.791 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/33 file after the snapshot 2024-05-27T06:48:59.793 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/34 file after the snapshot 2024-05-27T06:48:59.796 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/35 file after the snapshot 2024-05-27T06:48:59.798 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/36 file after the snapshot 2024-05-27T06:48:59.800 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/37 file after the snapshot 2024-05-27T06:48:59.803 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/38 file after the snapshot 2024-05-27T06:48:59.805 INFO:tasks.workunit.client.0.smithi081.stdout: create 233/39 file after the snapshot 2024-05-27T06:48:59.808 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/.snap/snap-subdir-test 2024-05-27T06:48:59.829 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/30 file after the snapshot 2024-05-27T06:48:59.862 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/31 file after the snapshot 2024-05-27T06:48:59.865 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/32 file after the snapshot 2024-05-27T06:48:59.867 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/33 file after the snapshot 2024-05-27T06:48:59.869 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/34 file after the snapshot 2024-05-27T06:48:59.872 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/35 file after the snapshot 2024-05-27T06:48:59.874 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/36 file after the snapshot 2024-05-27T06:48:59.877 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/37 file after the snapshot 2024-05-27T06:48:59.880 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/38 file after the snapshot 2024-05-27T06:48:59.883 INFO:tasks.workunit.client.0.smithi081.stdout: create 234/39 file after the snapshot 2024-05-27T06:48:59.885 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/.snap/snap-subdir-test 2024-05-27T06:48:59.925 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/30 file after the snapshot 2024-05-27T06:48:59.927 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/31 file after the snapshot 2024-05-27T06:48:59.930 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/32 file after the snapshot 2024-05-27T06:48:59.933 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/33 file after the snapshot 2024-05-27T06:48:59.935 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/34 file after the snapshot 2024-05-27T06:48:59.937 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/35 file after the snapshot 2024-05-27T06:48:59.939 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/36 file after the snapshot 2024-05-27T06:48:59.942 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/37 file after the snapshot 2024-05-27T06:48:59.945 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/38 file after the snapshot 2024-05-27T06:48:59.947 INFO:tasks.workunit.client.0.smithi081.stdout: create 235/39 file after the snapshot 2024-05-27T06:48:59.949 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/.snap/snap-subdir-test 2024-05-27T06:48:59.967 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/30 file after the snapshot 2024-05-27T06:48:59.969 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/31 file after the snapshot 2024-05-27T06:48:59.972 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/32 file after the snapshot 2024-05-27T06:48:59.974 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/33 file after the snapshot 2024-05-27T06:48:59.976 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/34 file after the snapshot 2024-05-27T06:48:59.979 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/35 file after the snapshot 2024-05-27T06:48:59.981 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/36 file after the snapshot 2024-05-27T06:48:59.983 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/37 file after the snapshot 2024-05-27T06:48:59.985 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/38 file after the snapshot 2024-05-27T06:48:59.988 INFO:tasks.workunit.client.0.smithi081.stdout: create 236/39 file after the snapshot 2024-05-27T06:48:59.990 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/.snap/snap-subdir-test 2024-05-27T06:49:00.012 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/30 file after the snapshot 2024-05-27T06:49:00.014 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/31 file after the snapshot 2024-05-27T06:49:00.016 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/32 file after the snapshot 2024-05-27T06:49:00.019 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/33 file after the snapshot 2024-05-27T06:49:00.021 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/34 file after the snapshot 2024-05-27T06:49:00.023 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/35 file after the snapshot 2024-05-27T06:49:00.025 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/36 file after the snapshot 2024-05-27T06:49:00.028 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/37 file after the snapshot 2024-05-27T06:49:00.031 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/38 file after the snapshot 2024-05-27T06:49:00.033 INFO:tasks.workunit.client.0.smithi081.stdout: create 237/39 file after the snapshot 2024-05-27T06:49:00.036 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/.snap/snap-subdir-test 2024-05-27T06:49:00.056 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/30 file after the snapshot 2024-05-27T06:49:00.058 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/31 file after the snapshot 2024-05-27T06:49:00.061 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/32 file after the snapshot 2024-05-27T06:49:00.063 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/33 file after the snapshot 2024-05-27T06:49:00.065 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/34 file after the snapshot 2024-05-27T06:49:00.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/35 file after the snapshot 2024-05-27T06:49:00.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/36 file after the snapshot 2024-05-27T06:49:00.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/37 file after the snapshot 2024-05-27T06:49:00.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/38 file after the snapshot 2024-05-27T06:49:00.078 INFO:tasks.workunit.client.0.smithi081.stdout: create 238/39 file after the snapshot 2024-05-27T06:49:00.080 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/.snap/snap-subdir-test 2024-05-27T06:49:00.098 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/30 file after the snapshot 2024-05-27T06:49:00.100 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/31 file after the snapshot 2024-05-27T06:49:00.103 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/32 file after the snapshot 2024-05-27T06:49:00.106 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/33 file after the snapshot 2024-05-27T06:49:00.108 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/34 file after the snapshot 2024-05-27T06:49:00.110 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/35 file after the snapshot 2024-05-27T06:49:00.113 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/36 file after the snapshot 2024-05-27T06:49:00.115 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/37 file after the snapshot 2024-05-27T06:49:00.118 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/38 file after the snapshot 2024-05-27T06:49:00.120 INFO:tasks.workunit.client.0.smithi081.stdout: create 239/39 file after the snapshot 2024-05-27T06:49:00.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/.snap/snap-subdir-test 2024-05-27T06:49:00.383 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/30 file after the snapshot 2024-05-27T06:49:00.385 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/31 file after the snapshot 2024-05-27T06:49:00.387 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/32 file after the snapshot 2024-05-27T06:49:00.390 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/33 file after the snapshot 2024-05-27T06:49:00.392 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/34 file after the snapshot 2024-05-27T06:49:00.394 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/35 file after the snapshot 2024-05-27T06:49:00.396 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/36 file after the snapshot 2024-05-27T06:49:00.399 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/37 file after the snapshot 2024-05-27T06:49:00.401 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/38 file after the snapshot 2024-05-27T06:49:00.404 INFO:tasks.workunit.client.0.smithi081.stdout: create 240/39 file after the snapshot 2024-05-27T06:49:00.407 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/.snap/snap-subdir-test 2024-05-27T06:49:00.426 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/30 file after the snapshot 2024-05-27T06:49:00.428 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/31 file after the snapshot 2024-05-27T06:49:00.431 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/32 file after the snapshot 2024-05-27T06:49:00.433 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/33 file after the snapshot 2024-05-27T06:49:00.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/34 file after the snapshot 2024-05-27T06:49:00.439 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/35 file after the snapshot 2024-05-27T06:49:00.441 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/36 file after the snapshot 2024-05-27T06:49:00.444 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/37 file after the snapshot 2024-05-27T06:49:00.446 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/38 file after the snapshot 2024-05-27T06:49:00.449 INFO:tasks.workunit.client.0.smithi081.stdout: create 241/39 file after the snapshot 2024-05-27T06:49:00.451 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/.snap/snap-subdir-test 2024-05-27T06:49:00.471 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/30 file after the snapshot 2024-05-27T06:49:00.473 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/31 file after the snapshot 2024-05-27T06:49:00.476 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/32 file after the snapshot 2024-05-27T06:49:00.478 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/33 file after the snapshot 2024-05-27T06:49:00.480 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/34 file after the snapshot 2024-05-27T06:49:00.482 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/35 file after the snapshot 2024-05-27T06:49:00.484 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/36 file after the snapshot 2024-05-27T06:49:00.487 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/37 file after the snapshot 2024-05-27T06:49:00.489 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/38 file after the snapshot 2024-05-27T06:49:00.491 INFO:tasks.workunit.client.0.smithi081.stdout: create 242/39 file after the snapshot 2024-05-27T06:49:00.494 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/.snap/snap-subdir-test 2024-05-27T06:49:00.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/30 file after the snapshot 2024-05-27T06:49:00.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/31 file after the snapshot 2024-05-27T06:49:00.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/32 file after the snapshot 2024-05-27T06:49:00.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/33 file after the snapshot 2024-05-27T06:49:00.522 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/34 file after the snapshot 2024-05-27T06:49:00.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/35 file after the snapshot 2024-05-27T06:49:00.527 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/36 file after the snapshot 2024-05-27T06:49:00.529 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/37 file after the snapshot 2024-05-27T06:49:00.531 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/38 file after the snapshot 2024-05-27T06:49:00.534 INFO:tasks.workunit.client.0.smithi081.stdout: create 243/39 file after the snapshot 2024-05-27T06:49:00.536 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/.snap/snap-subdir-test 2024-05-27T06:49:00.554 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/30 file after the snapshot 2024-05-27T06:49:00.556 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/31 file after the snapshot 2024-05-27T06:49:00.559 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/32 file after the snapshot 2024-05-27T06:49:00.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/33 file after the snapshot 2024-05-27T06:49:00.563 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/34 file after the snapshot 2024-05-27T06:49:00.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/35 file after the snapshot 2024-05-27T06:49:00.567 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/36 file after the snapshot 2024-05-27T06:49:00.570 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/37 file after the snapshot 2024-05-27T06:49:00.572 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/38 file after the snapshot 2024-05-27T06:49:00.574 INFO:tasks.workunit.client.0.smithi081.stdout: create 244/39 file after the snapshot 2024-05-27T06:49:00.576 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/.snap/snap-subdir-test 2024-05-27T06:49:00.594 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/30 file after the snapshot 2024-05-27T06:49:00.596 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/31 file after the snapshot 2024-05-27T06:49:00.598 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/32 file after the snapshot 2024-05-27T06:49:00.601 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/33 file after the snapshot 2024-05-27T06:49:00.603 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/34 file after the snapshot 2024-05-27T06:49:00.605 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/35 file after the snapshot 2024-05-27T06:49:00.607 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/36 file after the snapshot 2024-05-27T06:49:00.610 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/37 file after the snapshot 2024-05-27T06:49:00.612 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/38 file after the snapshot 2024-05-27T06:49:00.614 INFO:tasks.workunit.client.0.smithi081.stdout: create 245/39 file after the snapshot 2024-05-27T06:49:00.616 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/.snap/snap-subdir-test 2024-05-27T06:49:00.635 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/30 file after the snapshot 2024-05-27T06:49:00.637 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/31 file after the snapshot 2024-05-27T06:49:00.639 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/32 file after the snapshot 2024-05-27T06:49:00.642 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/33 file after the snapshot 2024-05-27T06:49:00.644 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/34 file after the snapshot 2024-05-27T06:49:00.647 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/35 file after the snapshot 2024-05-27T06:49:00.649 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/36 file after the snapshot 2024-05-27T06:49:00.652 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/37 file after the snapshot 2024-05-27T06:49:00.654 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/38 file after the snapshot 2024-05-27T06:49:00.657 INFO:tasks.workunit.client.0.smithi081.stdout: create 246/39 file after the snapshot 2024-05-27T06:49:00.659 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/.snap/snap-subdir-test 2024-05-27T06:49:00.678 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/30 file after the snapshot 2024-05-27T06:49:00.680 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/31 file after the snapshot 2024-05-27T06:49:00.683 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/32 file after the snapshot 2024-05-27T06:49:00.685 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/33 file after the snapshot 2024-05-27T06:49:00.688 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/34 file after the snapshot 2024-05-27T06:49:00.690 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/35 file after the snapshot 2024-05-27T06:49:00.693 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/36 file after the snapshot 2024-05-27T06:49:00.695 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/37 file after the snapshot 2024-05-27T06:49:00.697 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/38 file after the snapshot 2024-05-27T06:49:00.699 INFO:tasks.workunit.client.0.smithi081.stdout: create 247/39 file after the snapshot 2024-05-27T06:49:00.701 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/.snap/snap-subdir-test 2024-05-27T06:49:00.718 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/30 file after the snapshot 2024-05-27T06:49:00.720 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/31 file after the snapshot 2024-05-27T06:49:00.723 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/32 file after the snapshot 2024-05-27T06:49:00.726 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/33 file after the snapshot 2024-05-27T06:49:00.728 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/34 file after the snapshot 2024-05-27T06:49:00.731 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/35 file after the snapshot 2024-05-27T06:49:00.733 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/36 file after the snapshot 2024-05-27T06:49:00.736 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/37 file after the snapshot 2024-05-27T06:49:00.739 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/38 file after the snapshot 2024-05-27T06:49:00.741 INFO:tasks.workunit.client.0.smithi081.stdout: create 248/39 file after the snapshot 2024-05-27T06:49:00.744 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/.snap/snap-subdir-test 2024-05-27T06:49:00.762 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/30 file after the snapshot 2024-05-27T06:49:00.765 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/31 file after the snapshot 2024-05-27T06:49:00.768 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/32 file after the snapshot 2024-05-27T06:49:00.770 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/33 file after the snapshot 2024-05-27T06:49:00.773 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/34 file after the snapshot 2024-05-27T06:49:00.775 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/35 file after the snapshot 2024-05-27T06:49:00.778 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/36 file after the snapshot 2024-05-27T06:49:00.781 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/37 file after the snapshot 2024-05-27T06:49:00.784 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/38 file after the snapshot 2024-05-27T06:49:00.787 INFO:tasks.workunit.client.0.smithi081.stdout: create 249/39 file after the snapshot 2024-05-27T06:49:00.789 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/.snap/snap-subdir-test 2024-05-27T06:49:00.809 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/30 file after the snapshot 2024-05-27T06:49:00.811 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/31 file after the snapshot 2024-05-27T06:49:00.814 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/32 file after the snapshot 2024-05-27T06:49:00.816 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/33 file after the snapshot 2024-05-27T06:49:00.818 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/34 file after the snapshot 2024-05-27T06:49:00.820 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/35 file after the snapshot 2024-05-27T06:49:00.823 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/36 file after the snapshot 2024-05-27T06:49:00.825 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/37 file after the snapshot 2024-05-27T06:49:00.830 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/38 file after the snapshot 2024-05-27T06:49:00.839 INFO:tasks.workunit.client.0.smithi081.stdout: create 250/39 file after the snapshot 2024-05-27T06:49:00.841 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/.snap/snap-subdir-test 2024-05-27T06:49:00.866 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/30 file after the snapshot 2024-05-27T06:49:00.868 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/31 file after the snapshot 2024-05-27T06:49:00.871 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/32 file after the snapshot 2024-05-27T06:49:00.873 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/33 file after the snapshot 2024-05-27T06:49:00.875 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/34 file after the snapshot 2024-05-27T06:49:00.877 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/35 file after the snapshot 2024-05-27T06:49:00.880 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/36 file after the snapshot 2024-05-27T06:49:00.882 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/37 file after the snapshot 2024-05-27T06:49:00.885 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/38 file after the snapshot 2024-05-27T06:49:00.887 INFO:tasks.workunit.client.0.smithi081.stdout: create 251/39 file after the snapshot 2024-05-27T06:49:00.889 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/.snap/snap-subdir-test 2024-05-27T06:49:00.907 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/30 file after the snapshot 2024-05-27T06:49:00.909 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/31 file after the snapshot 2024-05-27T06:49:00.913 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/32 file after the snapshot 2024-05-27T06:49:00.914 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/33 file after the snapshot 2024-05-27T06:49:00.917 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/34 file after the snapshot 2024-05-27T06:49:00.919 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/35 file after the snapshot 2024-05-27T06:49:00.921 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/36 file after the snapshot 2024-05-27T06:49:00.923 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/37 file after the snapshot 2024-05-27T06:49:00.925 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/38 file after the snapshot 2024-05-27T06:49:00.927 INFO:tasks.workunit.client.0.smithi081.stdout: create 252/39 file after the snapshot 2024-05-27T06:49:00.929 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/.snap/snap-subdir-test 2024-05-27T06:49:00.948 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/30 file after the snapshot 2024-05-27T06:49:00.950 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/31 file after the snapshot 2024-05-27T06:49:00.952 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/32 file after the snapshot 2024-05-27T06:49:00.954 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/33 file after the snapshot 2024-05-27T06:49:00.956 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/34 file after the snapshot 2024-05-27T06:49:00.958 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/35 file after the snapshot 2024-05-27T06:49:00.961 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/36 file after the snapshot 2024-05-27T06:49:00.964 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/37 file after the snapshot 2024-05-27T06:49:00.966 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/38 file after the snapshot 2024-05-27T06:49:01.403 INFO:tasks.workunit.client.0.smithi081.stdout: create 253/39 file after the snapshot 2024-05-27T06:49:01.405 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/.snap/snap-subdir-test 2024-05-27T06:49:01.425 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/30 file after the snapshot 2024-05-27T06:49:01.428 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/31 file after the snapshot 2024-05-27T06:49:01.430 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/32 file after the snapshot 2024-05-27T06:49:01.433 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/33 file after the snapshot 2024-05-27T06:49:01.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/34 file after the snapshot 2024-05-27T06:49:01.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/35 file after the snapshot 2024-05-27T06:49:01.440 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/36 file after the snapshot 2024-05-27T06:49:01.442 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/37 file after the snapshot 2024-05-27T06:49:01.444 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/38 file after the snapshot 2024-05-27T06:49:01.447 INFO:tasks.workunit.client.0.smithi081.stdout: create 254/39 file after the snapshot 2024-05-27T06:49:01.449 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/.snap/snap-subdir-test 2024-05-27T06:49:01.468 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/30 file after the snapshot 2024-05-27T06:49:01.470 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/31 file after the snapshot 2024-05-27T06:49:01.472 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/32 file after the snapshot 2024-05-27T06:49:01.474 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/33 file after the snapshot 2024-05-27T06:49:01.476 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/34 file after the snapshot 2024-05-27T06:49:01.479 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/35 file after the snapshot 2024-05-27T06:49:01.481 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/36 file after the snapshot 2024-05-27T06:49:01.484 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/37 file after the snapshot 2024-05-27T06:49:01.486 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/38 file after the snapshot 2024-05-27T06:49:01.489 INFO:tasks.workunit.client.0.smithi081.stdout: create 255/39 file after the snapshot 2024-05-27T06:49:01.491 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/.snap/snap-subdir-test 2024-05-27T06:49:01.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/30 file after the snapshot 2024-05-27T06:49:01.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/31 file after the snapshot 2024-05-27T06:49:01.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/32 file after the snapshot 2024-05-27T06:49:01.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/33 file after the snapshot 2024-05-27T06:49:01.521 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/34 file after the snapshot 2024-05-27T06:49:01.523 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/35 file after the snapshot 2024-05-27T06:49:01.525 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/36 file after the snapshot 2024-05-27T06:49:01.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/37 file after the snapshot 2024-05-27T06:49:01.530 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/38 file after the snapshot 2024-05-27T06:49:01.532 INFO:tasks.workunit.client.0.smithi081.stdout: create 256/39 file after the snapshot 2024-05-27T06:49:01.534 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/.snap/snap-subdir-test 2024-05-27T06:49:01.552 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/30 file after the snapshot 2024-05-27T06:49:01.555 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/31 file after the snapshot 2024-05-27T06:49:01.557 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/32 file after the snapshot 2024-05-27T06:49:01.559 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/33 file after the snapshot 2024-05-27T06:49:01.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/34 file after the snapshot 2024-05-27T06:49:01.564 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/35 file after the snapshot 2024-05-27T06:49:01.566 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/36 file after the snapshot 2024-05-27T06:49:01.568 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/37 file after the snapshot 2024-05-27T06:49:01.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/38 file after the snapshot 2024-05-27T06:49:01.573 INFO:tasks.workunit.client.0.smithi081.stdout: create 257/39 file after the snapshot 2024-05-27T06:49:01.575 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/.snap/snap-subdir-test 2024-05-27T06:49:01.592 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/30 file after the snapshot 2024-05-27T06:49:01.594 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/31 file after the snapshot 2024-05-27T06:49:01.597 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/32 file after the snapshot 2024-05-27T06:49:01.599 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/33 file after the snapshot 2024-05-27T06:49:01.601 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/34 file after the snapshot 2024-05-27T06:49:01.604 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/35 file after the snapshot 2024-05-27T06:49:01.606 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/36 file after the snapshot 2024-05-27T06:49:01.608 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/37 file after the snapshot 2024-05-27T06:49:01.611 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/38 file after the snapshot 2024-05-27T06:49:01.613 INFO:tasks.workunit.client.0.smithi081.stdout: create 258/39 file after the snapshot 2024-05-27T06:49:01.615 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/.snap/snap-subdir-test 2024-05-27T06:49:01.634 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/30 file after the snapshot 2024-05-27T06:49:01.636 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/31 file after the snapshot 2024-05-27T06:49:01.639 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/32 file after the snapshot 2024-05-27T06:49:01.641 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/33 file after the snapshot 2024-05-27T06:49:01.644 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/34 file after the snapshot 2024-05-27T06:49:01.646 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/35 file after the snapshot 2024-05-27T06:49:01.648 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/36 file after the snapshot 2024-05-27T06:49:01.651 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/37 file after the snapshot 2024-05-27T06:49:01.653 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/38 file after the snapshot 2024-05-27T06:49:01.656 INFO:tasks.workunit.client.0.smithi081.stdout: create 259/39 file after the snapshot 2024-05-27T06:49:01.658 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/.snap/snap-subdir-test 2024-05-27T06:49:01.685 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/30 file after the snapshot 2024-05-27T06:49:01.687 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/31 file after the snapshot 2024-05-27T06:49:01.689 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/32 file after the snapshot 2024-05-27T06:49:01.693 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/33 file after the snapshot 2024-05-27T06:49:01.694 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/34 file after the snapshot 2024-05-27T06:49:01.696 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/35 file after the snapshot 2024-05-27T06:49:01.698 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/36 file after the snapshot 2024-05-27T06:49:01.701 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/37 file after the snapshot 2024-05-27T06:49:01.703 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/38 file after the snapshot 2024-05-27T06:49:01.705 INFO:tasks.workunit.client.0.smithi081.stdout: create 260/39 file after the snapshot 2024-05-27T06:49:01.707 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/.snap/snap-subdir-test 2024-05-27T06:49:01.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:49:01.992 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:49:02.509 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/30 file after the snapshot 2024-05-27T06:49:02.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/31 file after the snapshot 2024-05-27T06:49:02.514 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/32 file after the snapshot 2024-05-27T06:49:02.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/33 file after the snapshot 2024-05-27T06:49:02.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/34 file after the snapshot 2024-05-27T06:49:02.522 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/35 file after the snapshot 2024-05-27T06:49:02.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/36 file after the snapshot 2024-05-27T06:49:02.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/37 file after the snapshot 2024-05-27T06:49:02.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/38 file after the snapshot 2024-05-27T06:49:02.531 INFO:tasks.workunit.client.0.smithi081.stdout: create 261/39 file after the snapshot 2024-05-27T06:49:02.534 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/.snap/snap-subdir-test 2024-05-27T06:49:02.553 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/30 file after the snapshot 2024-05-27T06:49:02.556 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/31 file after the snapshot 2024-05-27T06:49:02.558 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/32 file after the snapshot 2024-05-27T06:49:02.835 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/33 file after the snapshot 2024-05-27T06:49:03.003 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/34 file after the snapshot 2024-05-27T06:49:03.006 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/35 file after the snapshot 2024-05-27T06:49:03.009 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/36 file after the snapshot 2024-05-27T06:49:03.011 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/37 file after the snapshot 2024-05-27T06:49:03.014 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/38 file after the snapshot 2024-05-27T06:49:03.016 INFO:tasks.workunit.client.0.smithi081.stdout: create 262/39 file after the snapshot 2024-05-27T06:49:03.019 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/.snap/snap-subdir-test 2024-05-27T06:49:03.049 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/30 file after the snapshot 2024-05-27T06:49:03.052 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/31 file after the snapshot 2024-05-27T06:49:03.054 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/32 file after the snapshot 2024-05-27T06:49:03.056 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/33 file after the snapshot 2024-05-27T06:49:03.058 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/34 file after the snapshot 2024-05-27T06:49:03.061 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/35 file after the snapshot 2024-05-27T06:49:03.063 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/36 file after the snapshot 2024-05-27T06:49:03.066 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/37 file after the snapshot 2024-05-27T06:49:03.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/38 file after the snapshot 2024-05-27T06:49:03.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 263/39 file after the snapshot 2024-05-27T06:49:03.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/.snap/snap-subdir-test 2024-05-27T06:49:03.091 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/30 file after the snapshot 2024-05-27T06:49:03.093 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/31 file after the snapshot 2024-05-27T06:49:03.096 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/32 file after the snapshot 2024-05-27T06:49:03.099 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/33 file after the snapshot 2024-05-27T06:49:03.101 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/34 file after the snapshot 2024-05-27T06:49:03.104 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/35 file after the snapshot 2024-05-27T06:49:03.106 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/36 file after the snapshot 2024-05-27T06:49:03.109 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/37 file after the snapshot 2024-05-27T06:49:03.111 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/38 file after the snapshot 2024-05-27T06:49:03.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 264/39 file after the snapshot 2024-05-27T06:49:03.116 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/.snap/snap-subdir-test 2024-05-27T06:49:03.135 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/30 file after the snapshot 2024-05-27T06:49:03.137 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/31 file after the snapshot 2024-05-27T06:49:03.140 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/32 file after the snapshot 2024-05-27T06:49:03.142 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/33 file after the snapshot 2024-05-27T06:49:03.144 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/34 file after the snapshot 2024-05-27T06:49:03.147 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/35 file after the snapshot 2024-05-27T06:49:03.149 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/36 file after the snapshot 2024-05-27T06:49:03.151 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/37 file after the snapshot 2024-05-27T06:49:03.154 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/38 file after the snapshot 2024-05-27T06:49:03.157 INFO:tasks.workunit.client.0.smithi081.stdout: create 265/39 file after the snapshot 2024-05-27T06:49:03.159 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/.snap/snap-subdir-test 2024-05-27T06:49:03.177 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/30 file after the snapshot 2024-05-27T06:49:03.179 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/31 file after the snapshot 2024-05-27T06:49:03.181 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/32 file after the snapshot 2024-05-27T06:49:03.183 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/33 file after the snapshot 2024-05-27T06:49:03.186 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/34 file after the snapshot 2024-05-27T06:49:03.188 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/35 file after the snapshot 2024-05-27T06:49:03.190 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/36 file after the snapshot 2024-05-27T06:49:03.192 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/37 file after the snapshot 2024-05-27T06:49:03.195 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/38 file after the snapshot 2024-05-27T06:49:03.197 INFO:tasks.workunit.client.0.smithi081.stdout: create 266/39 file after the snapshot 2024-05-27T06:49:03.200 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/.snap/snap-subdir-test 2024-05-27T06:49:03.465 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/30 file after the snapshot 2024-05-27T06:49:03.467 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/31 file after the snapshot 2024-05-27T06:49:03.470 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/32 file after the snapshot 2024-05-27T06:49:03.472 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/33 file after the snapshot 2024-05-27T06:49:03.475 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/34 file after the snapshot 2024-05-27T06:49:03.477 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/35 file after the snapshot 2024-05-27T06:49:03.479 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/36 file after the snapshot 2024-05-27T06:49:03.482 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/37 file after the snapshot 2024-05-27T06:49:03.484 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/38 file after the snapshot 2024-05-27T06:49:03.486 INFO:tasks.workunit.client.0.smithi081.stdout: create 267/39 file after the snapshot 2024-05-27T06:49:03.488 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/.snap/snap-subdir-test 2024-05-27T06:49:03.506 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/30 file after the snapshot 2024-05-27T06:49:03.508 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/31 file after the snapshot 2024-05-27T06:49:03.510 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/32 file after the snapshot 2024-05-27T06:49:03.513 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/33 file after the snapshot 2024-05-27T06:49:03.515 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/34 file after the snapshot 2024-05-27T06:49:03.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/35 file after the snapshot 2024-05-27T06:49:03.520 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/36 file after the snapshot 2024-05-27T06:49:03.522 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/37 file after the snapshot 2024-05-27T06:49:03.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/38 file after the snapshot 2024-05-27T06:49:03.527 INFO:tasks.workunit.client.0.smithi081.stdout: create 268/39 file after the snapshot 2024-05-27T06:49:03.529 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/.snap/snap-subdir-test 2024-05-27T06:49:03.547 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/30 file after the snapshot 2024-05-27T06:49:03.549 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/31 file after the snapshot 2024-05-27T06:49:03.551 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/32 file after the snapshot 2024-05-27T06:49:03.553 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/33 file after the snapshot 2024-05-27T06:49:03.556 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/34 file after the snapshot 2024-05-27T06:49:03.559 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/35 file after the snapshot 2024-05-27T06:49:03.561 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/36 file after the snapshot 2024-05-27T06:49:03.565 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/37 file after the snapshot 2024-05-27T06:49:03.566 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/38 file after the snapshot 2024-05-27T06:49:03.569 INFO:tasks.workunit.client.0.smithi081.stdout: create 269/39 file after the snapshot 2024-05-27T06:49:03.571 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/.snap/snap-subdir-test 2024-05-27T06:49:04.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/30 file after the snapshot 2024-05-27T06:49:04.309 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/31 file after the snapshot 2024-05-27T06:49:04.313 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/32 file after the snapshot 2024-05-27T06:49:04.316 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/33 file after the snapshot 2024-05-27T06:49:04.319 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/34 file after the snapshot 2024-05-27T06:49:04.321 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/35 file after the snapshot 2024-05-27T06:49:04.324 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/36 file after the snapshot 2024-05-27T06:49:04.327 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/37 file after the snapshot 2024-05-27T06:49:04.329 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/38 file after the snapshot 2024-05-27T06:49:04.332 INFO:tasks.workunit.client.0.smithi081.stdout: create 270/39 file after the snapshot 2024-05-27T06:49:04.334 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/.snap/snap-subdir-test 2024-05-27T06:49:04.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/30 file after the snapshot 2024-05-27T06:49:04.357 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/31 file after the snapshot 2024-05-27T06:49:04.360 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/32 file after the snapshot 2024-05-27T06:49:04.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/33 file after the snapshot 2024-05-27T06:49:04.365 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/34 file after the snapshot 2024-05-27T06:49:04.367 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/35 file after the snapshot 2024-05-27T06:49:04.370 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/36 file after the snapshot 2024-05-27T06:49:04.372 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/37 file after the snapshot 2024-05-27T06:49:04.375 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/38 file after the snapshot 2024-05-27T06:49:04.377 INFO:tasks.workunit.client.0.smithi081.stdout: create 271/39 file after the snapshot 2024-05-27T06:49:04.381 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/.snap/snap-subdir-test 2024-05-27T06:49:04.398 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/30 file after the snapshot 2024-05-27T06:49:04.400 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/31 file after the snapshot 2024-05-27T06:49:04.403 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/32 file after the snapshot 2024-05-27T06:49:04.406 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/33 file after the snapshot 2024-05-27T06:49:04.408 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/34 file after the snapshot 2024-05-27T06:49:04.410 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/35 file after the snapshot 2024-05-27T06:49:04.413 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/36 file after the snapshot 2024-05-27T06:49:04.415 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/37 file after the snapshot 2024-05-27T06:49:04.417 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/38 file after the snapshot 2024-05-27T06:49:04.420 INFO:tasks.workunit.client.0.smithi081.stdout: create 272/39 file after the snapshot 2024-05-27T06:49:04.422 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/.snap/snap-subdir-test 2024-05-27T06:49:04.441 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/30 file after the snapshot 2024-05-27T06:49:04.443 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/31 file after the snapshot 2024-05-27T06:49:04.445 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/32 file after the snapshot 2024-05-27T06:49:04.448 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/33 file after the snapshot 2024-05-27T06:49:04.450 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/34 file after the snapshot 2024-05-27T06:49:04.453 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/35 file after the snapshot 2024-05-27T06:49:04.456 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/36 file after the snapshot 2024-05-27T06:49:04.459 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/37 file after the snapshot 2024-05-27T06:49:04.461 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/38 file after the snapshot 2024-05-27T06:49:04.464 INFO:tasks.workunit.client.0.smithi081.stdout: create 273/39 file after the snapshot 2024-05-27T06:49:04.466 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/.snap/snap-subdir-test 2024-05-27T06:49:04.486 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/30 file after the snapshot 2024-05-27T06:49:04.488 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/31 file after the snapshot 2024-05-27T06:49:04.491 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/32 file after the snapshot 2024-05-27T06:49:04.493 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/33 file after the snapshot 2024-05-27T06:49:04.495 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/34 file after the snapshot 2024-05-27T06:49:04.497 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/35 file after the snapshot 2024-05-27T06:49:04.500 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/36 file after the snapshot 2024-05-27T06:49:04.502 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/37 file after the snapshot 2024-05-27T06:49:04.504 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/38 file after the snapshot 2024-05-27T06:49:04.507 INFO:tasks.workunit.client.0.smithi081.stdout: create 274/39 file after the snapshot 2024-05-27T06:49:04.509 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/.snap/snap-subdir-test 2024-05-27T06:49:04.527 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/30 file after the snapshot 2024-05-27T06:49:04.529 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/31 file after the snapshot 2024-05-27T06:49:04.532 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/32 file after the snapshot 2024-05-27T06:49:04.535 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/33 file after the snapshot 2024-05-27T06:49:04.537 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/34 file after the snapshot 2024-05-27T06:49:04.539 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/35 file after the snapshot 2024-05-27T06:49:04.542 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/36 file after the snapshot 2024-05-27T06:49:04.545 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/37 file after the snapshot 2024-05-27T06:49:04.547 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/38 file after the snapshot 2024-05-27T06:49:04.549 INFO:tasks.workunit.client.0.smithi081.stdout: create 275/39 file after the snapshot 2024-05-27T06:49:04.552 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/.snap/snap-subdir-test 2024-05-27T06:49:05.509 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/30 file after the snapshot 2024-05-27T06:49:05.512 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/31 file after the snapshot 2024-05-27T06:49:05.515 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/32 file after the snapshot 2024-05-27T06:49:05.517 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/33 file after the snapshot 2024-05-27T06:49:05.520 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/34 file after the snapshot 2024-05-27T06:49:05.523 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/35 file after the snapshot 2024-05-27T06:49:05.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/36 file after the snapshot 2024-05-27T06:49:05.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/37 file after the snapshot 2024-05-27T06:49:05.531 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/38 file after the snapshot 2024-05-27T06:49:05.533 INFO:tasks.workunit.client.0.smithi081.stdout: create 276/39 file after the snapshot 2024-05-27T06:49:05.536 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/.snap/snap-subdir-test 2024-05-27T06:49:05.939 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/30 file after the snapshot 2024-05-27T06:49:05.941 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/31 file after the snapshot 2024-05-27T06:49:05.944 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/32 file after the snapshot 2024-05-27T06:49:05.946 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/33 file after the snapshot 2024-05-27T06:49:05.949 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/34 file after the snapshot 2024-05-27T06:49:05.952 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/35 file after the snapshot 2024-05-27T06:49:05.954 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/36 file after the snapshot 2024-05-27T06:49:05.957 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/37 file after the snapshot 2024-05-27T06:49:05.959 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/38 file after the snapshot 2024-05-27T06:49:05.962 INFO:tasks.workunit.client.0.smithi081.stdout: create 277/39 file after the snapshot 2024-05-27T06:49:05.965 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/.snap/snap-subdir-test 2024-05-27T06:49:06.049 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/30 file after the snapshot 2024-05-27T06:49:06.051 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/31 file after the snapshot 2024-05-27T06:49:06.054 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/32 file after the snapshot 2024-05-27T06:49:06.057 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/33 file after the snapshot 2024-05-27T06:49:06.059 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/34 file after the snapshot 2024-05-27T06:49:06.062 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/35 file after the snapshot 2024-05-27T06:49:06.065 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/36 file after the snapshot 2024-05-27T06:49:06.068 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/37 file after the snapshot 2024-05-27T06:49:06.070 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/38 file after the snapshot 2024-05-27T06:49:06.073 INFO:tasks.workunit.client.0.smithi081.stdout: create 278/39 file after the snapshot 2024-05-27T06:49:06.075 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/.snap/snap-subdir-test 2024-05-27T06:49:06.097 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/30 file after the snapshot 2024-05-27T06:49:06.099 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/31 file after the snapshot 2024-05-27T06:49:06.102 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/32 file after the snapshot 2024-05-27T06:49:06.104 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/33 file after the snapshot 2024-05-27T06:49:06.107 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/34 file after the snapshot 2024-05-27T06:49:06.109 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/35 file after the snapshot 2024-05-27T06:49:06.112 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/36 file after the snapshot 2024-05-27T06:49:06.114 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/37 file after the snapshot 2024-05-27T06:49:06.117 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/38 file after the snapshot 2024-05-27T06:49:06.120 INFO:tasks.workunit.client.0.smithi081.stdout: create 279/39 file after the snapshot 2024-05-27T06:49:06.123 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/.snap/snap-subdir-test 2024-05-27T06:49:06.143 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/30 file after the snapshot 2024-05-27T06:49:06.145 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/31 file after the snapshot 2024-05-27T06:49:06.148 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/32 file after the snapshot 2024-05-27T06:49:06.150 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/33 file after the snapshot 2024-05-27T06:49:06.153 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/34 file after the snapshot 2024-05-27T06:49:06.155 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/35 file after the snapshot 2024-05-27T06:49:06.158 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/36 file after the snapshot 2024-05-27T06:49:06.161 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/37 file after the snapshot 2024-05-27T06:49:06.163 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/38 file after the snapshot 2024-05-27T06:49:06.165 INFO:tasks.workunit.client.0.smithi081.stdout: create 280/39 file after the snapshot 2024-05-27T06:49:06.168 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/.snap/snap-subdir-test 2024-05-27T06:49:06.186 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/30 file after the snapshot 2024-05-27T06:49:06.188 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/31 file after the snapshot 2024-05-27T06:49:06.190 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/32 file after the snapshot 2024-05-27T06:49:06.192 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/33 file after the snapshot 2024-05-27T06:49:06.194 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/34 file after the snapshot 2024-05-27T06:49:06.196 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/35 file after the snapshot 2024-05-27T06:49:06.199 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/36 file after the snapshot 2024-05-27T06:49:06.201 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/37 file after the snapshot 2024-05-27T06:49:06.203 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/38 file after the snapshot 2024-05-27T06:49:06.205 INFO:tasks.workunit.client.0.smithi081.stdout: create 281/39 file after the snapshot 2024-05-27T06:49:06.208 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/.snap/snap-subdir-test 2024-05-27T06:49:06.226 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/30 file after the snapshot 2024-05-27T06:49:06.228 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/31 file after the snapshot 2024-05-27T06:49:06.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/32 file after the snapshot 2024-05-27T06:49:06.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/33 file after the snapshot 2024-05-27T06:49:06.235 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/34 file after the snapshot 2024-05-27T06:49:06.237 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/35 file after the snapshot 2024-05-27T06:49:06.239 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/36 file after the snapshot 2024-05-27T06:49:06.241 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/37 file after the snapshot 2024-05-27T06:49:06.244 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/38 file after the snapshot 2024-05-27T06:49:06.246 INFO:tasks.workunit.client.0.smithi081.stdout: create 282/39 file after the snapshot 2024-05-27T06:49:06.248 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/.snap/snap-subdir-test 2024-05-27T06:49:06.266 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/30 file after the snapshot 2024-05-27T06:49:06.269 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/31 file after the snapshot 2024-05-27T06:49:06.271 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/32 file after the snapshot 2024-05-27T06:49:06.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/33 file after the snapshot 2024-05-27T06:49:06.275 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/34 file after the snapshot 2024-05-27T06:49:06.277 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/35 file after the snapshot 2024-05-27T06:49:06.280 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/36 file after the snapshot 2024-05-27T06:49:06.282 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/37 file after the snapshot 2024-05-27T06:49:06.284 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/38 file after the snapshot 2024-05-27T06:49:06.286 INFO:tasks.workunit.client.0.smithi081.stdout: create 283/39 file after the snapshot 2024-05-27T06:49:06.288 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/.snap/snap-subdir-test 2024-05-27T06:49:06.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/30 file after the snapshot 2024-05-27T06:49:06.308 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/31 file after the snapshot 2024-05-27T06:49:06.311 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/32 file after the snapshot 2024-05-27T06:49:06.313 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/33 file after the snapshot 2024-05-27T06:49:06.316 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/34 file after the snapshot 2024-05-27T06:49:06.318 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/35 file after the snapshot 2024-05-27T06:49:06.320 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/36 file after the snapshot 2024-05-27T06:49:06.322 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/37 file after the snapshot 2024-05-27T06:49:06.325 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/38 file after the snapshot 2024-05-27T06:49:06.328 INFO:tasks.workunit.client.0.smithi081.stdout: create 284/39 file after the snapshot 2024-05-27T06:49:06.330 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/.snap/snap-subdir-test 2024-05-27T06:49:06.350 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/30 file after the snapshot 2024-05-27T06:49:06.352 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/31 file after the snapshot 2024-05-27T06:49:06.354 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/32 file after the snapshot 2024-05-27T06:49:06.356 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/33 file after the snapshot 2024-05-27T06:49:06.358 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/34 file after the snapshot 2024-05-27T06:49:06.361 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/35 file after the snapshot 2024-05-27T06:49:06.363 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/36 file after the snapshot 2024-05-27T06:49:06.365 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/37 file after the snapshot 2024-05-27T06:49:06.367 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/38 file after the snapshot 2024-05-27T06:49:06.370 INFO:tasks.workunit.client.0.smithi081.stdout: create 285/39 file after the snapshot 2024-05-27T06:49:06.372 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/.snap/snap-subdir-test 2024-05-27T06:49:06.391 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/30 file after the snapshot 2024-05-27T06:49:06.393 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/31 file after the snapshot 2024-05-27T06:49:06.395 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/32 file after the snapshot 2024-05-27T06:49:06.398 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/33 file after the snapshot 2024-05-27T06:49:06.401 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/34 file after the snapshot 2024-05-27T06:49:06.402 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/35 file after the snapshot 2024-05-27T06:49:06.404 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/36 file after the snapshot 2024-05-27T06:49:06.406 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/37 file after the snapshot 2024-05-27T06:49:06.408 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/38 file after the snapshot 2024-05-27T06:49:06.411 INFO:tasks.workunit.client.0.smithi081.stdout: create 286/39 file after the snapshot 2024-05-27T06:49:06.413 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/.snap/snap-subdir-test 2024-05-27T06:49:06.433 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/30 file after the snapshot 2024-05-27T06:49:06.434 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/31 file after the snapshot 2024-05-27T06:49:06.436 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/32 file after the snapshot 2024-05-27T06:49:06.438 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/33 file after the snapshot 2024-05-27T06:49:06.441 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/34 file after the snapshot 2024-05-27T06:49:06.444 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/35 file after the snapshot 2024-05-27T06:49:06.446 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/36 file after the snapshot 2024-05-27T06:49:06.448 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/37 file after the snapshot 2024-05-27T06:49:06.451 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/38 file after the snapshot 2024-05-27T06:49:06.453 INFO:tasks.workunit.client.0.smithi081.stdout: create 287/39 file after the snapshot 2024-05-27T06:49:06.455 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/.snap/snap-subdir-test 2024-05-27T06:49:06.473 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/30 file after the snapshot 2024-05-27T06:49:06.475 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/31 file after the snapshot 2024-05-27T06:49:06.478 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/32 file after the snapshot 2024-05-27T06:49:06.481 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/33 file after the snapshot 2024-05-27T06:49:06.483 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/34 file after the snapshot 2024-05-27T06:49:06.485 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/35 file after the snapshot 2024-05-27T06:49:06.488 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/36 file after the snapshot 2024-05-27T06:49:06.490 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/37 file after the snapshot 2024-05-27T06:49:06.492 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/38 file after the snapshot 2024-05-27T06:49:06.495 INFO:tasks.workunit.client.0.smithi081.stdout: create 288/39 file after the snapshot 2024-05-27T06:49:06.497 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/.snap/snap-subdir-test 2024-05-27T06:49:06.516 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/30 file after the snapshot 2024-05-27T06:49:06.519 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/31 file after the snapshot 2024-05-27T06:49:06.522 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/32 file after the snapshot 2024-05-27T06:49:06.524 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/33 file after the snapshot 2024-05-27T06:49:06.526 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/34 file after the snapshot 2024-05-27T06:49:06.528 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/35 file after the snapshot 2024-05-27T06:49:06.531 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/36 file after the snapshot 2024-05-27T06:49:06.532 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/37 file after the snapshot 2024-05-27T06:49:06.535 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/38 file after the snapshot 2024-05-27T06:49:06.537 INFO:tasks.workunit.client.0.smithi081.stdout: create 289/39 file after the snapshot 2024-05-27T06:49:06.539 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/.snap/snap-subdir-test 2024-05-27T06:49:07.119 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/30 file after the snapshot 2024-05-27T06:49:07.121 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/31 file after the snapshot 2024-05-27T06:49:07.124 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/32 file after the snapshot 2024-05-27T06:49:07.126 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/33 file after the snapshot 2024-05-27T06:49:07.129 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/34 file after the snapshot 2024-05-27T06:49:07.132 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/35 file after the snapshot 2024-05-27T06:49:07.134 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/36 file after the snapshot 2024-05-27T06:49:07.136 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/37 file after the snapshot 2024-05-27T06:49:07.138 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/38 file after the snapshot 2024-05-27T06:49:07.141 INFO:tasks.workunit.client.0.smithi081.stdout: create 290/39 file after the snapshot 2024-05-27T06:49:07.143 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/.snap/snap-subdir-test 2024-05-27T06:49:07.163 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/30 file after the snapshot 2024-05-27T06:49:07.166 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/31 file after the snapshot 2024-05-27T06:49:07.168 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/32 file after the snapshot 2024-05-27T06:49:07.170 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/33 file after the snapshot 2024-05-27T06:49:07.173 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/34 file after the snapshot 2024-05-27T06:49:07.175 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/35 file after the snapshot 2024-05-27T06:49:07.178 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/36 file after the snapshot 2024-05-27T06:49:07.180 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/37 file after the snapshot 2024-05-27T06:49:07.182 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/38 file after the snapshot 2024-05-27T06:49:07.185 INFO:tasks.workunit.client.0.smithi081.stdout: create 291/39 file after the snapshot 2024-05-27T06:49:07.187 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/.snap/snap-subdir-test 2024-05-27T06:49:07.210 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/30 file after the snapshot 2024-05-27T06:49:07.212 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/31 file after the snapshot 2024-05-27T06:49:07.215 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/32 file after the snapshot 2024-05-27T06:49:07.217 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/33 file after the snapshot 2024-05-27T06:49:07.219 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/34 file after the snapshot 2024-05-27T06:49:07.221 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/35 file after the snapshot 2024-05-27T06:49:07.223 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/36 file after the snapshot 2024-05-27T06:49:07.226 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/37 file after the snapshot 2024-05-27T06:49:07.228 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/38 file after the snapshot 2024-05-27T06:49:07.230 INFO:tasks.workunit.client.0.smithi081.stdout: create 292/39 file after the snapshot 2024-05-27T06:49:07.232 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/.snap/snap-subdir-test 2024-05-27T06:49:07.251 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/30 file after the snapshot 2024-05-27T06:49:07.253 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/31 file after the snapshot 2024-05-27T06:49:07.255 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/32 file after the snapshot 2024-05-27T06:49:07.257 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/33 file after the snapshot 2024-05-27T06:49:07.259 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/34 file after the snapshot 2024-05-27T06:49:07.262 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/35 file after the snapshot 2024-05-27T06:49:07.264 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/36 file after the snapshot 2024-05-27T06:49:07.266 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/37 file after the snapshot 2024-05-27T06:49:07.268 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/38 file after the snapshot 2024-05-27T06:49:07.271 INFO:tasks.workunit.client.0.smithi081.stdout: create 293/39 file after the snapshot 2024-05-27T06:49:07.273 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/.snap/snap-subdir-test 2024-05-27T06:49:07.291 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/30 file after the snapshot 2024-05-27T06:49:07.293 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/31 file after the snapshot 2024-05-27T06:49:07.295 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/32 file after the snapshot 2024-05-27T06:49:07.298 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/33 file after the snapshot 2024-05-27T06:49:07.300 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/34 file after the snapshot 2024-05-27T06:49:07.303 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/35 file after the snapshot 2024-05-27T06:49:07.305 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/36 file after the snapshot 2024-05-27T06:49:07.307 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/37 file after the snapshot 2024-05-27T06:49:07.310 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/38 file after the snapshot 2024-05-27T06:49:07.312 INFO:tasks.workunit.client.0.smithi081.stdout: create 294/39 file after the snapshot 2024-05-27T06:49:07.314 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/.snap/snap-subdir-test 2024-05-27T06:49:07.332 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/30 file after the snapshot 2024-05-27T06:49:07.334 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/31 file after the snapshot 2024-05-27T06:49:07.337 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/32 file after the snapshot 2024-05-27T06:49:07.339 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/33 file after the snapshot 2024-05-27T06:49:07.342 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/34 file after the snapshot 2024-05-27T06:49:07.344 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/35 file after the snapshot 2024-05-27T06:49:07.346 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/36 file after the snapshot 2024-05-27T06:49:07.348 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/37 file after the snapshot 2024-05-27T06:49:07.350 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/38 file after the snapshot 2024-05-27T06:49:07.353 INFO:tasks.workunit.client.0.smithi081.stdout: create 295/39 file after the snapshot 2024-05-27T06:49:07.355 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/.snap/snap-subdir-test 2024-05-27T06:49:07.373 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/30 file after the snapshot 2024-05-27T06:49:07.375 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/31 file after the snapshot 2024-05-27T06:49:07.377 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/32 file after the snapshot 2024-05-27T06:49:07.379 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/33 file after the snapshot 2024-05-27T06:49:07.382 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/34 file after the snapshot 2024-05-27T06:49:07.384 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/35 file after the snapshot 2024-05-27T06:49:07.387 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/36 file after the snapshot 2024-05-27T06:49:07.389 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/37 file after the snapshot 2024-05-27T06:49:07.391 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/38 file after the snapshot 2024-05-27T06:49:07.393 INFO:tasks.workunit.client.0.smithi081.stdout: create 296/39 file after the snapshot 2024-05-27T06:49:07.395 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/.snap/snap-subdir-test 2024-05-27T06:49:07.415 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/30 file after the snapshot 2024-05-27T06:49:07.417 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/31 file after the snapshot 2024-05-27T06:49:07.420 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/32 file after the snapshot 2024-05-27T06:49:07.422 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/33 file after the snapshot 2024-05-27T06:49:07.424 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/34 file after the snapshot 2024-05-27T06:49:07.427 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/35 file after the snapshot 2024-05-27T06:49:07.429 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/36 file after the snapshot 2024-05-27T06:49:07.432 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/37 file after the snapshot 2024-05-27T06:49:07.434 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/38 file after the snapshot 2024-05-27T06:49:07.437 INFO:tasks.workunit.client.0.smithi081.stdout: create 297/39 file after the snapshot 2024-05-27T06:49:07.439 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/.snap/snap-subdir-test 2024-05-27T06:49:07.460 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/30 file after the snapshot 2024-05-27T06:49:07.462 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/31 file after the snapshot 2024-05-27T06:49:07.465 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/32 file after the snapshot 2024-05-27T06:49:07.467 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/33 file after the snapshot 2024-05-27T06:49:07.470 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/34 file after the snapshot 2024-05-27T06:49:07.472 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/35 file after the snapshot 2024-05-27T06:49:07.474 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/36 file after the snapshot 2024-05-27T06:49:07.477 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/37 file after the snapshot 2024-05-27T06:49:07.479 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/38 file after the snapshot 2024-05-27T06:49:07.481 INFO:tasks.workunit.client.0.smithi081.stdout: create 298/39 file after the snapshot 2024-05-27T06:49:07.483 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/.snap/snap-subdir-test 2024-05-27T06:49:08.362 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/30 file after the snapshot 2024-05-27T06:49:08.364 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/31 file after the snapshot 2024-05-27T06:49:08.367 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/32 file after the snapshot 2024-05-27T06:49:08.370 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/33 file after the snapshot 2024-05-27T06:49:08.372 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/34 file after the snapshot 2024-05-27T06:49:08.375 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/35 file after the snapshot 2024-05-27T06:49:08.378 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/36 file after the snapshot 2024-05-27T06:49:08.380 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/37 file after the snapshot 2024-05-27T06:49:08.383 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/38 file after the snapshot 2024-05-27T06:49:08.385 INFO:tasks.workunit.client.0.smithi081.stdout: create 299/39 file after the snapshot 2024-05-27T06:49:08.388 INFO:tasks.workunit.client.0.smithi081.stdout:Take second snapshot .snap/test2 2024-05-27T06:49:08.475 INFO:tasks.workunit.client.0.smithi081.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-05-27T06:49:13.751 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-05-27T06:49:13.751 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-05-27T06:49:13.751 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-05-27T06:49:13.751 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-05-27T06:49:13.751 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-05-27T06:49:13.752 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-05-27T06:49:13.753 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-05-27T06:49:13.754 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-05-27T06:49:13.755 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-05-27T06:49:13.756 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-05-27T06:49:13.757 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-05-27T06:49:15.787 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-05-27T06:49:15.787 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-05-27T06:49:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-05-27T06:49:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-05-27T06:49:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-05-27T06:49:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-05-27T06:49:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-05-27T06:49:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-05-27T06:49:19.515 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-05-27T06:49:19.515 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-05-27T06:49:19.515 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-05-27T06:49:19.515 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-05-27T06:49:19.515 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-05-27T06:49:19.516 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-05-27T06:49:19.517 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-05-27T06:49:19.518 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-05-27T06:49:19.519 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-05-27T06:49:19.520 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-05-27T06:49:19.521 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-05-27T06:49:22.568 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-05-27T06:49:22.569 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-05-27T06:49:22.569 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-05-27T06:49:22.569 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-05-27T06:49:22.570 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-05-27T06:49:22.570 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-05-27T06:49:22.570 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-05-27T06:49:22.570 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-05-27T06:49:22.570 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-05-27T06:49:22.571 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-05-27T06:49:22.571 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-05-27T06:49:22.571 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-05-27T06:49:22.571 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-05-27T06:49:22.571 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-05-27T06:49:22.572 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-05-27T06:49:22.573 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-05-27T06:49:22.574 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-05-27T06:49:22.575 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-05-27T06:49:22.576 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-05-27T06:49:22.577 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-05-27T06:49:22.578 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-05-27T06:49:22.578 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-05-27T06:49:22.578 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-05-27T06:49:26.034 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-05-27T06:49:26.035 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-05-27T06:49:26.036 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-05-27T06:49:26.037 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-05-27T06:49:26.038 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-05-27T06:49:26.039 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-05-27T06:49:26.040 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-05-27T06:49:26.041 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-05-27T06:49:29.389 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-05-27T06:49:29.389 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-05-27T06:49:29.389 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-05-27T06:49:29.390 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-05-27T06:49:29.391 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-05-27T06:49:29.392 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-05-27T06:49:29.393 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-05-27T06:49:29.394 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-05-27T06:49:29.395 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-05-27T06:49:29.396 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-05-27T06:49:29.396 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-05-27T06:49:29.396 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-05-27T06:49:29.396 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-05-27T06:49:32.045 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:49:32.049 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-05-27T06:49:32.228 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-05-27T06:49:32.229 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-05-27T06:49:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-05-27T06:49:32.231 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-05-27T06:49:32.232 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-05-27T06:49:32.233 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-05-27T06:49:32.234 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-05-27T06:49:32.234 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-05-27T06:49:34.507 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-05-27T06:49:34.508 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-05-27T06:49:34.509 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-05-27T06:49:34.510 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-05-27T06:49:34.511 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-05-27T06:49:34.512 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-05-27T06:49:34.513 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-05-27T06:49:36.382 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-05-27T06:49:36.383 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-05-27T06:49:36.384 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-05-27T06:49:36.385 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-05-27T06:49:36.386 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-05-27T06:49:36.387 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-05-27T06:49:36.388 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-05-27T06:49:38.981 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-05-27T06:49:38.981 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-05-27T06:49:38.981 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-05-27T06:49:38.982 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-05-27T06:49:38.983 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-05-27T06:49:38.984 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-05-27T06:49:38.985 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-05-27T06:49:38.986 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-05-27T06:49:38.987 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-05-27T06:49:40.181 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-05-27T06:49:40.182 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-05-27T06:49:40.183 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-05-27T06:49:40.184 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-05-27T06:49:40.185 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-05-27T06:49:40.186 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-05-27T06:49:40.187 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-05-27T06:49:41.703 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-05-27T06:49:41.704 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-05-27T06:49:41.705 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-05-27T06:49:41.706 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-05-27T06:49:41.707 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-05-27T06:49:41.708 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-05-27T06:49:41.709 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-05-27T06:49:44.074 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-05-27T06:49:44.075 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-05-27T06:49:44.076 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-05-27T06:49:44.077 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-05-27T06:49:44.078 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-05-27T06:49:44.079 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-05-27T06:49:44.080 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-05-27T06:49:47.119 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-05-27T06:49:47.120 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-05-27T06:49:47.121 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-05-27T06:49:47.122 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-05-27T06:49:47.123 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-05-27T06:49:47.124 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-05-27T06:49:47.125 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-05-27T06:49:47.126 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-05-27T06:49:47.126 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-05-27T06:49:50.907 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-05-27T06:49:50.907 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-05-27T06:49:50.907 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-05-27T06:49:50.907 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-05-27T06:49:50.908 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-05-27T06:49:50.909 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-05-27T06:49:50.910 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-05-27T06:49:50.911 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-05-27T06:49:50.912 INFO:tasks.workunit.client.0.smithi081.stdout:Take third snapshot .snap/test3 2024-05-27T06:49:51.058 INFO:tasks.workunit.client.0.smithi081.stdout:Delete the snapshots... 2024-05-27T06:49:54.256 INFO:tasks.workunit.client.0.smithi081.stdout:.//.snap/test1 2024-05-27T06:49:54.256 INFO:tasks.workunit.client.0.smithi081.stdout:.//.snap/test2 2024-05-27T06:49:54.256 INFO:tasks.workunit.client.0.smithi081.stdout:.//.snap/test3 2024-05-27T06:49:54.274 INFO:tasks.workunit.client.0.smithi081.stdout:./274/.snap/snap-subdir-test 2024-05-27T06:49:54.293 INFO:tasks.workunit.client.0.smithi081.stdout:./166/.snap/snap-subdir-test 2024-05-27T06:49:54.309 INFO:tasks.workunit.client.0.smithi081.stdout:./256/.snap/snap-subdir-test 2024-05-27T06:49:54.323 INFO:tasks.workunit.client.0.smithi081.stdout:./247/.snap/snap-subdir-test 2024-05-27T06:49:54.338 INFO:tasks.workunit.client.0.smithi081.stdout:./248/.snap/snap-subdir-test 2024-05-27T06:49:54.355 INFO:tasks.workunit.client.0.smithi081.stdout:./147/.snap/snap-subdir-test 2024-05-27T06:49:54.370 INFO:tasks.workunit.client.0.smithi081.stdout:./151/.snap/snap-subdir-test 2024-05-27T06:49:54.386 INFO:tasks.workunit.client.0.smithi081.stdout:./170/.snap/snap-subdir-test 2024-05-27T06:49:54.401 INFO:tasks.workunit.client.0.smithi081.stdout:./167/.snap/snap-subdir-test 2024-05-27T06:49:54.417 INFO:tasks.workunit.client.0.smithi081.stdout:./140/.snap/snap-subdir-test 2024-05-27T06:49:54.435 INFO:tasks.workunit.client.0.smithi081.stdout:./142/.snap/snap-subdir-test 2024-05-27T06:49:54.452 INFO:tasks.workunit.client.0.smithi081.stdout:./150/.snap/snap-subdir-test 2024-05-27T06:49:54.467 INFO:tasks.workunit.client.0.smithi081.stdout:./273/.snap/snap-subdir-test 2024-05-27T06:49:54.482 INFO:tasks.workunit.client.0.smithi081.stdout:./269/.snap/snap-subdir-test 2024-05-27T06:49:54.500 INFO:tasks.workunit.client.0.smithi081.stdout:./105/.snap/snap-subdir-test 2024-05-27T06:49:54.520 INFO:tasks.workunit.client.0.smithi081.stdout:./210/.snap/snap-subdir-test 2024-05-27T06:49:54.538 INFO:tasks.workunit.client.0.smithi081.stdout:./265/.snap/snap-subdir-test 2024-05-27T06:49:54.556 INFO:tasks.workunit.client.0.smithi081.stdout:./218/.snap/snap-subdir-test 2024-05-27T06:49:54.576 INFO:tasks.workunit.client.0.smithi081.stdout:./290/.snap/snap-subdir-test 2024-05-27T06:49:54.596 INFO:tasks.workunit.client.0.smithi081.stdout:./149/.snap/snap-subdir-test 2024-05-27T06:49:55.390 INFO:tasks.workunit.client.0.smithi081.stdout:./109/.snap/snap-subdir-test 2024-05-27T06:49:55.408 INFO:tasks.workunit.client.0.smithi081.stdout:./292/.snap/snap-subdir-test 2024-05-27T06:49:56.022 INFO:tasks.workunit.client.0.smithi081.stdout:./242/.snap/snap-subdir-test 2024-05-27T06:49:56.041 INFO:tasks.workunit.client.0.smithi081.stdout:./158/.snap/snap-subdir-test 2024-05-27T06:49:56.058 INFO:tasks.workunit.client.0.smithi081.stdout:./243/.snap/snap-subdir-test 2024-05-27T06:49:56.076 INFO:tasks.workunit.client.0.smithi081.stdout:./106/.snap/snap-subdir-test 2024-05-27T06:49:56.092 INFO:tasks.workunit.client.0.smithi081.stdout:./250/.snap/snap-subdir-test 2024-05-27T06:49:56.108 INFO:tasks.workunit.client.0.smithi081.stdout:./221/.snap/snap-subdir-test 2024-05-27T06:49:56.125 INFO:tasks.workunit.client.0.smithi081.stdout:./244/.snap/snap-subdir-test 2024-05-27T06:49:56.141 INFO:tasks.workunit.client.0.smithi081.stdout:./258/.snap/snap-subdir-test 2024-05-27T06:49:56.504 INFO:tasks.workunit.client.0.smithi081.stdout:./268/.snap/snap-subdir-test 2024-05-27T06:49:56.523 INFO:tasks.workunit.client.0.smithi081.stdout:./189/.snap/snap-subdir-test 2024-05-27T06:49:56.541 INFO:tasks.workunit.client.0.smithi081.stdout:./285/.snap/snap-subdir-test 2024-05-27T06:49:56.559 INFO:tasks.workunit.client.0.smithi081.stdout:./208/.snap/snap-subdir-test 2024-05-27T06:49:56.574 INFO:tasks.workunit.client.0.smithi081.stdout:./288/.snap/snap-subdir-test 2024-05-27T06:49:56.590 INFO:tasks.workunit.client.0.smithi081.stdout:./209/.snap/snap-subdir-test 2024-05-27T06:49:56.606 INFO:tasks.workunit.client.0.smithi081.stdout:./253/.snap/snap-subdir-test 2024-05-27T06:49:56.622 INFO:tasks.workunit.client.0.smithi081.stdout:./249/.snap/snap-subdir-test 2024-05-27T06:49:56.637 INFO:tasks.workunit.client.0.smithi081.stdout:./133/.snap/snap-subdir-test 2024-05-27T06:49:56.653 INFO:tasks.workunit.client.0.smithi081.stdout:./291/.snap/snap-subdir-test 2024-05-27T06:49:57.667 INFO:tasks.workunit.client.0.smithi081.stdout:./226/.snap/snap-subdir-test 2024-05-27T06:49:57.687 INFO:tasks.workunit.client.0.smithi081.stdout:./173/.snap/snap-subdir-test 2024-05-27T06:49:57.706 INFO:tasks.workunit.client.0.smithi081.stdout:./174/.snap/snap-subdir-test 2024-05-27T06:49:57.727 INFO:tasks.workunit.client.0.smithi081.stdout:./299/.snap/snap-subdir-test 2024-05-27T06:49:57.746 INFO:tasks.workunit.client.0.smithi081.stdout:./262/.snap/snap-subdir-test 2024-05-27T06:49:57.766 INFO:tasks.workunit.client.0.smithi081.stdout:./231/.snap/snap-subdir-test 2024-05-27T06:49:58.040 INFO:tasks.workunit.client.0.smithi081.stdout:./214/.snap/snap-subdir-test 2024-05-27T06:49:58.060 INFO:tasks.workunit.client.0.smithi081.stdout:./281/.snap/snap-subdir-test 2024-05-27T06:49:58.080 INFO:tasks.workunit.client.0.smithi081.stdout:./289/.snap/snap-subdir-test 2024-05-27T06:49:58.098 INFO:tasks.workunit.client.0.smithi081.stdout:./283/.snap/snap-subdir-test 2024-05-27T06:49:58.117 INFO:tasks.workunit.client.0.smithi081.stdout:./232/.snap/snap-subdir-test 2024-05-27T06:49:58.136 INFO:tasks.workunit.client.0.smithi081.stdout:./298/.snap/snap-subdir-test 2024-05-27T06:49:58.154 INFO:tasks.workunit.client.0.smithi081.stdout:./233/.snap/snap-subdir-test 2024-05-27T06:49:58.173 INFO:tasks.workunit.client.0.smithi081.stdout:./155/.snap/snap-subdir-test 2024-05-27T06:49:58.190 INFO:tasks.workunit.client.0.smithi081.stdout:./114/.snap/snap-subdir-test 2024-05-27T06:49:58.517 INFO:tasks.workunit.client.0.smithi081.stdout:./159/.snap/snap-subdir-test 2024-05-27T06:49:58.537 INFO:tasks.workunit.client.0.smithi081.stdout:./187/.snap/snap-subdir-test 2024-05-27T06:49:58.559 INFO:tasks.workunit.client.0.smithi081.stdout:./185/.snap/snap-subdir-test 2024-05-27T06:49:58.577 INFO:tasks.workunit.client.0.smithi081.stdout:./293/.snap/snap-subdir-test 2024-05-27T06:49:58.593 INFO:tasks.workunit.client.0.smithi081.stdout:./228/.snap/snap-subdir-test 2024-05-27T06:49:58.610 INFO:tasks.workunit.client.0.smithi081.stdout:./120/.snap/snap-subdir-test 2024-05-27T06:49:58.627 INFO:tasks.workunit.client.0.smithi081.stdout:./110/.snap/snap-subdir-test 2024-05-27T06:49:58.645 INFO:tasks.workunit.client.0.smithi081.stdout:./267/.snap/snap-subdir-test 2024-05-27T06:49:58.660 INFO:tasks.workunit.client.0.smithi081.stdout:./202/.snap/snap-subdir-test 2024-05-27T06:49:58.678 INFO:tasks.workunit.client.0.smithi081.stdout:./137/.snap/snap-subdir-test 2024-05-27T06:49:58.695 INFO:tasks.workunit.client.0.smithi081.stdout:./246/.snap/snap-subdir-test 2024-05-27T06:49:58.711 INFO:tasks.workunit.client.0.smithi081.stdout:./219/.snap/snap-subdir-test 2024-05-27T06:49:58.728 INFO:tasks.workunit.client.0.smithi081.stdout:./198/.snap/snap-subdir-test 2024-05-27T06:49:59.377 INFO:tasks.workunit.client.0.smithi081.stdout:./136/.snap/snap-subdir-test 2024-05-27T06:49:59.399 INFO:tasks.workunit.client.0.smithi081.stdout:./270/.snap/snap-subdir-test 2024-05-27T06:49:59.419 INFO:tasks.workunit.client.0.smithi081.stdout:./125/.snap/snap-subdir-test 2024-05-27T06:49:59.440 INFO:tasks.workunit.client.0.smithi081.stdout:./194/.snap/snap-subdir-test 2024-05-27T06:49:59.460 INFO:tasks.workunit.client.0.smithi081.stdout:./119/.snap/snap-subdir-test 2024-05-27T06:49:59.478 INFO:tasks.workunit.client.0.smithi081.stdout:./297/.snap/snap-subdir-test 2024-05-27T06:49:59.495 INFO:tasks.workunit.client.0.smithi081.stdout:./230/.snap/snap-subdir-test 2024-05-27T06:49:59.513 INFO:tasks.workunit.client.0.smithi081.stdout:./260/.snap/snap-subdir-test 2024-05-27T06:49:59.534 INFO:tasks.workunit.client.0.smithi081.stdout:./157/.snap/snap-subdir-test 2024-05-27T06:49:59.553 INFO:tasks.workunit.client.0.smithi081.stdout:./172/.snap/snap-subdir-test 2024-05-27T06:49:59.572 INFO:tasks.workunit.client.0.smithi081.stdout:./277/.snap/snap-subdir-test 2024-05-27T06:49:59.593 INFO:tasks.workunit.client.0.smithi081.stdout:./186/.snap/snap-subdir-test 2024-05-27T06:49:59.611 INFO:tasks.workunit.client.0.smithi081.stdout:./237/.snap/snap-subdir-test 2024-05-27T06:49:59.629 INFO:tasks.workunit.client.0.smithi081.stdout:./190/.snap/snap-subdir-test 2024-05-27T06:49:59.643 INFO:tasks.workunit.client.0.smithi081.stdout:./213/.snap/snap-subdir-test 2024-05-27T06:49:59.658 INFO:tasks.workunit.client.0.smithi081.stdout:./104/.snap/snap-subdir-test 2024-05-27T06:49:59.676 INFO:tasks.workunit.client.0.smithi081.stdout:./164/.snap/snap-subdir-test 2024-05-27T06:49:59.695 INFO:tasks.workunit.client.0.smithi081.stdout:./112/.snap/snap-subdir-test 2024-05-27T06:49:59.711 INFO:tasks.workunit.client.0.smithi081.stdout:./295/.snap/snap-subdir-test 2024-05-27T06:49:59.728 INFO:tasks.workunit.client.0.smithi081.stdout:./272/.snap/snap-subdir-test 2024-05-27T06:49:59.747 INFO:tasks.workunit.client.0.smithi081.stdout:./100/.snap/snap-subdir-test 2024-05-27T06:49:59.765 INFO:tasks.workunit.client.0.smithi081.stdout:./134/.snap/snap-subdir-test 2024-05-27T06:49:59.782 INFO:tasks.workunit.client.0.smithi081.stdout:./200/.snap/snap-subdir-test 2024-05-27T06:49:59.800 INFO:tasks.workunit.client.0.smithi081.stdout:./115/.snap/snap-subdir-test 2024-05-27T06:49:59.815 INFO:tasks.workunit.client.0.smithi081.stdout:./284/.snap/snap-subdir-test 2024-05-27T06:49:59.830 INFO:tasks.workunit.client.0.smithi081.stdout:./280/.snap/snap-subdir-test 2024-05-27T06:49:59.848 INFO:tasks.workunit.client.0.smithi081.stdout:./116/.snap/snap-subdir-test 2024-05-27T06:49:59.865 INFO:tasks.workunit.client.0.smithi081.stdout:./245/.snap/snap-subdir-test 2024-05-27T06:49:59.882 INFO:tasks.workunit.client.0.smithi081.stdout:./195/.snap/snap-subdir-test 2024-05-27T06:49:59.898 INFO:tasks.workunit.client.0.smithi081.stdout:./138/.snap/snap-subdir-test 2024-05-27T06:49:59.916 INFO:tasks.workunit.client.0.smithi081.stdout:./135/.snap/snap-subdir-test 2024-05-27T06:49:59.933 INFO:tasks.workunit.client.0.smithi081.stdout:./117/.snap/snap-subdir-test 2024-05-27T06:49:59.949 INFO:tasks.workunit.client.0.smithi081.stdout:./241/.snap/snap-subdir-test 2024-05-27T06:49:59.967 INFO:tasks.workunit.client.0.smithi081.stdout:./143/.snap/snap-subdir-test 2024-05-27T06:49:59.983 INFO:tasks.workunit.client.0.smithi081.stdout:./180/.snap/snap-subdir-test 2024-05-27T06:50:00.394 INFO:tasks.workunit.client.0.smithi081.stdout:./126/.snap/snap-subdir-test 2024-05-27T06:50:00.415 INFO:tasks.workunit.client.0.smithi081.stdout:./161/.snap/snap-subdir-test 2024-05-27T06:50:00.432 INFO:tasks.workunit.client.0.smithi081.stdout:./204/.snap/snap-subdir-test 2024-05-27T06:50:00.451 INFO:tasks.workunit.client.0.smithi081.stdout:./176/.snap/snap-subdir-test 2024-05-27T06:50:00.466 INFO:tasks.workunit.client.0.smithi081.stdout:./255/.snap/snap-subdir-test 2024-05-27T06:50:00.481 INFO:tasks.workunit.client.0.smithi081.stdout:./215/.snap/snap-subdir-test 2024-05-27T06:50:00.498 INFO:tasks.workunit.client.0.smithi081.stdout:./111/.snap/snap-subdir-test 2024-05-27T06:50:00.513 INFO:tasks.workunit.client.0.smithi081.stdout:./238/.snap/snap-subdir-test 2024-05-27T06:50:00.530 INFO:tasks.workunit.client.0.smithi081.stdout:./191/.snap/snap-subdir-test 2024-05-27T06:50:00.549 INFO:tasks.workunit.client.0.smithi081.stdout:./184/.snap/snap-subdir-test 2024-05-27T06:50:00.568 INFO:tasks.workunit.client.0.smithi081.stdout:./192/.snap/snap-subdir-test 2024-05-27T06:50:00.587 INFO:tasks.workunit.client.0.smithi081.stdout:./103/.snap/snap-subdir-test 2024-05-27T06:50:00.606 INFO:tasks.workunit.client.0.smithi081.stdout:./128/.snap/snap-subdir-test 2024-05-27T06:50:00.622 INFO:tasks.workunit.client.0.smithi081.stdout:./229/.snap/snap-subdir-test 2024-05-27T06:50:00.641 INFO:tasks.workunit.client.0.smithi081.stdout:./144/.snap/snap-subdir-test 2024-05-27T06:50:01.595 INFO:tasks.workunit.client.0.smithi081.stdout:./146/.snap/snap-subdir-test 2024-05-27T06:50:01.615 INFO:tasks.workunit.client.0.smithi081.stdout:./227/.snap/snap-subdir-test 2024-05-27T06:50:01.634 INFO:tasks.workunit.client.0.smithi081.stdout:./278/.snap/snap-subdir-test 2024-05-27T06:50:01.654 INFO:tasks.workunit.client.0.smithi081.stdout:./141/.snap/snap-subdir-test 2024-05-27T06:50:01.671 INFO:tasks.workunit.client.0.smithi081.stdout:./234/.snap/snap-subdir-test 2024-05-27T06:50:01.690 INFO:tasks.workunit.client.0.smithi081.stdout:./225/.snap/snap-subdir-test 2024-05-27T06:50:01.708 INFO:tasks.workunit.client.0.smithi081.stdout:./182/.snap/snap-subdir-test 2024-05-27T06:50:01.728 INFO:tasks.workunit.client.0.smithi081.stdout:./107/.snap/snap-subdir-test 2024-05-27T06:50:01.746 INFO:tasks.workunit.client.0.smithi081.stdout:./178/.snap/snap-subdir-test 2024-05-27T06:50:01.764 INFO:tasks.workunit.client.0.smithi081.stdout:./266/.snap/snap-subdir-test 2024-05-27T06:50:01.782 INFO:tasks.workunit.client.0.smithi081.stdout:./240/.snap/snap-subdir-test 2024-05-27T06:50:01.800 INFO:tasks.workunit.client.0.smithi081.stdout:./156/.snap/snap-subdir-test 2024-05-27T06:50:01.817 INFO:tasks.workunit.client.0.smithi081.stdout:./131/.snap/snap-subdir-test 2024-05-27T06:50:01.833 INFO:tasks.workunit.client.0.smithi081.stdout:./123/.snap/snap-subdir-test 2024-05-27T06:50:01.849 INFO:tasks.workunit.client.0.smithi081.stdout:./160/.snap/snap-subdir-test 2024-05-27T06:50:01.865 INFO:tasks.workunit.client.0.smithi081.stdout:./165/.snap/snap-subdir-test 2024-05-27T06:50:01.882 INFO:tasks.workunit.client.0.smithi081.stdout:./101/.snap/snap-subdir-test 2024-05-27T06:50:01.899 INFO:tasks.workunit.client.0.smithi081.stdout:./148/.snap/snap-subdir-test 2024-05-27T06:50:01.917 INFO:tasks.workunit.client.0.smithi081.stdout:./168/.snap/snap-subdir-test 2024-05-27T06:50:01.934 INFO:tasks.workunit.client.0.smithi081.stdout:./179/.snap/snap-subdir-test 2024-05-27T06:50:01.950 INFO:tasks.workunit.client.0.smithi081.stdout:./257/.snap/snap-subdir-test 2024-05-27T06:50:01.965 INFO:tasks.workunit.client.0.smithi081.stdout:./252/.snap/snap-subdir-test 2024-05-27T06:50:01.981 INFO:tasks.workunit.client.0.smithi081.stdout:./132/.snap/snap-subdir-test 2024-05-27T06:50:01.997 INFO:tasks.workunit.client.0.smithi081.stdout:./236/.snap/snap-subdir-test 2024-05-27T06:50:02.013 INFO:tasks.workunit.client.0.smithi081.stdout:./211/.snap/snap-subdir-test 2024-05-27T06:50:02.028 INFO:tasks.workunit.client.0.smithi081.stdout:./199/.snap/snap-subdir-test 2024-05-27T06:50:02.044 INFO:tasks.workunit.client.0.smithi081.stdout:./205/.snap/snap-subdir-test 2024-05-27T06:50:02.062 INFO:tasks.workunit.client.0.smithi081.stdout:./163/.snap/snap-subdir-test 2024-05-27T06:50:02.077 INFO:tasks.workunit.client.0.smithi081.stdout:./261/.snap/snap-subdir-test 2024-05-27T06:50:02.093 INFO:tasks.workunit.client.0.smithi081.stdout:./201/.snap/snap-subdir-test 2024-05-27T06:50:02.110 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:50:02.110 INFO:tasks.workunit.client.0.smithi081.stdout:./282/.snap/snap-subdir-test 2024-05-27T06:50:02.113 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:50:02.125 INFO:tasks.workunit.client.0.smithi081.stdout:./296/.snap/snap-subdir-test 2024-05-27T06:50:02.140 INFO:tasks.workunit.client.0.smithi081.stdout:./217/.snap/snap-subdir-test 2024-05-27T06:50:02.155 INFO:tasks.workunit.client.0.smithi081.stdout:./153/.snap/snap-subdir-test 2024-05-27T06:50:02.170 INFO:tasks.workunit.client.0.smithi081.stdout:./206/.snap/snap-subdir-test 2024-05-27T06:50:02.187 INFO:tasks.workunit.client.0.smithi081.stdout:./122/.snap/snap-subdir-test 2024-05-27T06:50:02.203 INFO:tasks.workunit.client.0.smithi081.stdout:./162/.snap/snap-subdir-test 2024-05-27T06:50:02.220 INFO:tasks.workunit.client.0.smithi081.stdout:./197/.snap/snap-subdir-test 2024-05-27T06:50:02.233 INFO:tasks.workunit.client.0.smithi081.stdout:./216/.snap/snap-subdir-test 2024-05-27T06:50:04.107 INFO:tasks.workunit.client.0.smithi081.stdout:./254/.snap/snap-subdir-test 2024-05-27T06:50:04.122 INFO:tasks.workunit.client.0.smithi081.stdout:./113/.snap/snap-subdir-test 2024-05-27T06:50:04.141 INFO:tasks.workunit.client.0.smithi081.stdout:./152/.snap/snap-subdir-test 2024-05-27T06:50:04.156 INFO:tasks.workunit.client.0.smithi081.stdout:./121/.snap/snap-subdir-test 2024-05-27T06:50:04.174 INFO:tasks.workunit.client.0.smithi081.stdout:./118/.snap/snap-subdir-test 2024-05-27T06:50:04.193 INFO:tasks.workunit.client.0.smithi081.stdout:./196/.snap/snap-subdir-test 2024-05-27T06:50:04.208 INFO:tasks.workunit.client.0.smithi081.stdout:./124/.snap/snap-subdir-test 2024-05-27T06:50:04.224 INFO:tasks.workunit.client.0.smithi081.stdout:./239/.snap/snap-subdir-test 2024-05-27T06:50:04.241 INFO:tasks.workunit.client.0.smithi081.stdout:./188/.snap/snap-subdir-test 2024-05-27T06:50:05.135 INFO:tasks.workunit.client.0.smithi081.stdout:./287/.snap/snap-subdir-test 2024-05-27T06:50:05.159 INFO:tasks.workunit.client.0.smithi081.stdout:./275/.snap/snap-subdir-test 2024-05-27T06:50:05.176 INFO:tasks.workunit.client.0.smithi081.stdout:./259/.snap/snap-subdir-test 2024-05-27T06:50:05.812 INFO:tasks.workunit.client.0.smithi081.stdout:./224/.snap/snap-subdir-test 2024-05-27T06:50:05.833 INFO:tasks.workunit.client.0.smithi081.stdout:./264/.snap/snap-subdir-test 2024-05-27T06:50:05.853 INFO:tasks.workunit.client.0.smithi081.stdout:./145/.snap/snap-subdir-test 2024-05-27T06:50:05.872 INFO:tasks.workunit.client.0.smithi081.stdout:./279/.snap/snap-subdir-test 2024-05-27T06:50:05.890 INFO:tasks.workunit.client.0.smithi081.stdout:./263/.snap/snap-subdir-test 2024-05-27T06:50:05.909 INFO:tasks.workunit.client.0.smithi081.stdout:./130/.snap/snap-subdir-test 2024-05-27T06:50:05.928 INFO:tasks.workunit.client.0.smithi081.stdout:./193/.snap/snap-subdir-test 2024-05-27T06:50:05.945 INFO:tasks.workunit.client.0.smithi081.stdout:./223/.snap/snap-subdir-test 2024-05-27T06:50:05.962 INFO:tasks.workunit.client.0.smithi081.stdout:./207/.snap/snap-subdir-test 2024-05-27T06:50:05.979 INFO:tasks.workunit.client.0.smithi081.stdout:./276/.snap/snap-subdir-test 2024-05-27T06:50:05.996 INFO:tasks.workunit.client.0.smithi081.stdout:./271/.snap/snap-subdir-test 2024-05-27T06:50:06.496 INFO:tasks.workunit.client.0.smithi081.stdout:./251/.snap/snap-subdir-test 2024-05-27T06:50:06.514 INFO:tasks.workunit.client.0.smithi081.stdout:./175/.snap/snap-subdir-test 2024-05-27T06:50:06.530 INFO:tasks.workunit.client.0.smithi081.stdout:./222/.snap/snap-subdir-test 2024-05-27T06:50:06.547 INFO:tasks.workunit.client.0.smithi081.stdout:./220/.snap/snap-subdir-test 2024-05-27T06:50:06.563 INFO:tasks.workunit.client.0.smithi081.stdout:./203/.snap/snap-subdir-test 2024-05-27T06:50:06.582 INFO:tasks.workunit.client.0.smithi081.stdout:./181/.snap/snap-subdir-test 2024-05-27T06:50:06.595 INFO:tasks.workunit.client.0.smithi081.stdout:./286/.snap/snap-subdir-test 2024-05-27T06:50:06.612 INFO:tasks.workunit.client.0.smithi081.stdout:./127/.snap/snap-subdir-test 2024-05-27T06:50:06.628 INFO:tasks.workunit.client.0.smithi081.stdout:./108/.snap/snap-subdir-test 2024-05-27T06:50:06.646 INFO:tasks.workunit.client.0.smithi081.stdout:./183/.snap/snap-subdir-test 2024-05-27T06:50:06.659 INFO:tasks.workunit.client.0.smithi081.stdout:./294/.snap/snap-subdir-test 2024-05-27T06:50:06.677 INFO:tasks.workunit.client.0.smithi081.stdout:./235/.snap/snap-subdir-test 2024-05-27T06:50:06.693 INFO:tasks.workunit.client.0.smithi081.stdout:./177/.snap/snap-subdir-test 2024-05-27T06:50:06.708 INFO:tasks.workunit.client.0.smithi081.stdout:./169/.snap/snap-subdir-test 2024-05-27T06:50:06.724 INFO:tasks.workunit.client.0.smithi081.stdout:./212/.snap/snap-subdir-test 2024-05-27T06:50:06.738 INFO:tasks.workunit.client.0.smithi081.stdout:./154/.snap/snap-subdir-test 2024-05-27T06:50:06.754 INFO:tasks.workunit.client.0.smithi081.stdout:./171/.snap/snap-subdir-test 2024-05-27T06:50:06.769 INFO:tasks.workunit.client.0.smithi081.stdout:./139/.snap/snap-subdir-test 2024-05-27T06:50:07.218 INFO:tasks.workunit.client.0.smithi081.stdout:./102/.snap/snap-subdir-test 2024-05-27T06:50:07.238 INFO:tasks.workunit.client.0.smithi081.stdout:./129/.snap/snap-subdir-test 2024-05-27T06:50:21.240 INFO:tasks.workunit.client.0.smithi081.stdout:Delete all the files and directories ... 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/19' 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/15' 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/31' 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/36' 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/20' 2024-05-27T06:50:23.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/12' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/39' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/10' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/34' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/38' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/11' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/16' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/32' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/30' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/14' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/18' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/13' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/33' 2024-05-27T06:50:23.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/17' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/35' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './100/37' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './100' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/19' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/15' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/31' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/36' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/20' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/12' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/39' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/10' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/34' 2024-05-27T06:50:23.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/38' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/11' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/16' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/32' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/30' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/14' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/18' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/13' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/33' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/17' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/35' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './101/37' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './101' 2024-05-27T06:50:23.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/19' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/15' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/31' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/36' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/20' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/12' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/39' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/10' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/34' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/38' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/11' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/16' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/32' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/30' 2024-05-27T06:50:23.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/14' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/18' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/13' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/33' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/17' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/35' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './102/37' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './102' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/19' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/15' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/31' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/36' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/20' 2024-05-27T06:50:23.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/12' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/39' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/10' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/34' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/38' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/11' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/16' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/32' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/30' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/14' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/18' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/13' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/33' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/17' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/35' 2024-05-27T06:50:23.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './103/37' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './103' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/19' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/15' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/31' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/36' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/20' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/12' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/39' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/10' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/34' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/38' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/11' 2024-05-27T06:50:23.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/16' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/32' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/30' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/14' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/18' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/13' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/33' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/17' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/35' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './104/37' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './104' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/19' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/15' 2024-05-27T06:50:23.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/31' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/36' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/20' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/12' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/39' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/10' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/34' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/38' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/11' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/16' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/32' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/30' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/14' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/18' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/13' 2024-05-27T06:50:23.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/33' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/17' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/35' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './105/37' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './105' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/19' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/15' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/31' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/36' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/20' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/12' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/39' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/10' 2024-05-27T06:50:23.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/34' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/38' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/11' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/16' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/32' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/30' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/14' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/18' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/13' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/33' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/17' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/35' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './106/37' 2024-05-27T06:50:23.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './106' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/19' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/15' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/31' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/36' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/20' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/12' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/39' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/10' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/34' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/38' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/11' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/16' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/32' 2024-05-27T06:50:23.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/30' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/14' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/18' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/13' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/33' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/17' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/35' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './107/37' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './107' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/19' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/15' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/31' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/36' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/20' 2024-05-27T06:50:23.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/12' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/39' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/10' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/34' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/38' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/11' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/16' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/32' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/30' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/14' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/18' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/13' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/33' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/17' 2024-05-27T06:50:23.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/35' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './108/37' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './108' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/19' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/15' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/31' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/36' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/20' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/12' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/39' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/10' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/34' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/38' 2024-05-27T06:50:23.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/11' 2024-05-27T06:50:23.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/16' 2024-05-27T06:50:23.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/32' 2024-05-27T06:50:23.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/30' 2024-05-27T06:50:25.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/14' 2024-05-27T06:50:25.023 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/18' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/13' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/33' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/17' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/35' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './109/37' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './109' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/19' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/15' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/31' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/36' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/20' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/12' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/39' 2024-05-27T06:50:25.024 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/10' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/34' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/38' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/11' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/16' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/32' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/30' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/14' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/18' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/13' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/33' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/17' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/35' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed './110/37' 2024-05-27T06:50:25.025 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './110' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/19' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/15' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/31' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/36' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/20' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/12' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/39' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/10' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/34' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/38' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/11' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/16' 2024-05-27T06:50:25.026 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/32' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/30' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/14' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/18' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/13' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/33' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/17' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/35' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './111/37' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './111' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/19' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/15' 2024-05-27T06:50:25.027 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/31' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/36' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/20' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/12' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/39' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/10' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/34' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/38' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/11' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/16' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/32' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/30' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/14' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/18' 2024-05-27T06:50:25.028 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/13' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/33' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/17' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/35' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './112/37' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './112' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/19' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/15' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/31' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/36' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/20' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/12' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/39' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/10' 2024-05-27T06:50:25.029 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/34' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/38' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/11' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/16' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/32' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/30' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/14' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/18' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/13' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/33' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/17' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/35' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed './113/37' 2024-05-27T06:50:25.030 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './113' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/19' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/15' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/31' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/36' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/20' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/12' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/39' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/10' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/34' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/38' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/11' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/16' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/32' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/30' 2024-05-27T06:50:25.031 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/14' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/18' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/13' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/33' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/17' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/35' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './114/37' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './114' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/19' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/15' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/31' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/36' 2024-05-27T06:50:25.032 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/20' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/12' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/39' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/10' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/34' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/38' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/11' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/16' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/32' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/30' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/14' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/18' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/13' 2024-05-27T06:50:25.033 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/33' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/17' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/35' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './115/37' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './115' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/19' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/15' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/31' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/36' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/20' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/12' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/39' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/10' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/34' 2024-05-27T06:50:25.034 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/38' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/11' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/16' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/32' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/30' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/14' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/18' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/13' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/33' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/17' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/35' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './116/37' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './116' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/19' 2024-05-27T06:50:25.035 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/15' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/31' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/36' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/20' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/12' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/39' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/10' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/34' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/38' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/11' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/16' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/32' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/30' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/14' 2024-05-27T06:50:25.036 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/18' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/13' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/33' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/17' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/35' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './117/37' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './117' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/19' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/15' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/31' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/36' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/20' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/12' 2024-05-27T06:50:25.037 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/39' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/10' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/34' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/38' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/11' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/16' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/32' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/30' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/14' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/18' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/13' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/33' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/17' 2024-05-27T06:50:25.038 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/35' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './118/37' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './118' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/19' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/15' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/31' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/36' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/20' 2024-05-27T06:50:25.039 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/12' 2024-05-27T06:50:25.641 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/39' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/10' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/34' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/38' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/11' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/16' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/32' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/30' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/14' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/18' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/13' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/33' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/17' 2024-05-27T06:50:25.642 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/35' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './119/37' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './119' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/19' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/15' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/31' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/36' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/20' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/12' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/39' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/10' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/34' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/38' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/11' 2024-05-27T06:50:25.643 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/16' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/32' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/30' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/14' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/18' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/13' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/33' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/17' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/35' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './120/37' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './120' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/19' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/15' 2024-05-27T06:50:25.644 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/31' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/36' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/20' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/12' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/39' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/10' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/34' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/38' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/11' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/16' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/32' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/30' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/14' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/18' 2024-05-27T06:50:25.645 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/13' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/33' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/17' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/35' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './121/37' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './121' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/19' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/15' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/31' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/36' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/20' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/12' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/39' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/10' 2024-05-27T06:50:25.646 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/34' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/38' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/11' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/16' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/32' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/30' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/14' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/18' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/13' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/33' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/17' 2024-05-27T06:50:25.647 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/35' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './122/37' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './122' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/19' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/15' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/31' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/36' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/20' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/12' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/39' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/10' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/34' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/38' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/11' 2024-05-27T06:50:25.648 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/16' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/32' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/30' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/14' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/18' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/13' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/33' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/17' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/35' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './123/37' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './123' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/19' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/15' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/31' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/36' 2024-05-27T06:50:25.649 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/20' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/12' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/39' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/10' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/34' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/38' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/11' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/16' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/32' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/30' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/14' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/18' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/13' 2024-05-27T06:50:25.650 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/33' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/17' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/35' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './124/37' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './124' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/19' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/15' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/31' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/36' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/20' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/12' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/39' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/10' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/34' 2024-05-27T06:50:25.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/38' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/11' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/16' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/32' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/30' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/14' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/18' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/13' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/33' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/17' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/35' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './125/37' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './125' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/19' 2024-05-27T06:50:25.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/15' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/31' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/36' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/20' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/12' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/39' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/10' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/34' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/38' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/11' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/16' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/32' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/30' 2024-05-27T06:50:25.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/14' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/18' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/13' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/33' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/17' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/35' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './126/37' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './126' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/19' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/15' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/31' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/36' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/20' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/12' 2024-05-27T06:50:25.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/39' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/10' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/34' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/38' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/11' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/16' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/32' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/30' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/14' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/18' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/13' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/33' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/17' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/35' 2024-05-27T06:50:25.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './127/37' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './127' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/19' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/15' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/31' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/36' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/20' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/12' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/39' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/10' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/34' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/38' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/11' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/16' 2024-05-27T06:50:25.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/32' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/30' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/14' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/18' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/13' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/33' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/17' 2024-05-27T06:50:25.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/35' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './128/37' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './128' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/19' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/15' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/31' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/36' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/20' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/12' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/39' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/10' 2024-05-27T06:50:27.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/34' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/38' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/11' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/16' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/32' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/30' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/14' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/18' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/13' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/33' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/17' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/35' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './129/37' 2024-05-27T06:50:27.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './129' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/19' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/15' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/31' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/36' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/20' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/12' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/39' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/10' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/34' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/38' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/11' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/16' 2024-05-27T06:50:27.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/32' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/30' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/14' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/18' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/13' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/33' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/17' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/35' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './130/37' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './130' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/19' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/15' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/31' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/36' 2024-05-27T06:50:27.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/20' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/12' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/39' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/10' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/34' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/38' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/11' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/16' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/32' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/30' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/14' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/18' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/13' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/33' 2024-05-27T06:50:27.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/17' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/35' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './131/37' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './131' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/19' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/15' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/31' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/36' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/20' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/12' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/39' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/10' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/34' 2024-05-27T06:50:27.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/38' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/11' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/16' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/32' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/30' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/14' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/18' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/13' 2024-05-27T06:50:27.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/33' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/17' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/35' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './132/37' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './132' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/19' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/15' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/31' 2024-05-27T06:50:27.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/36' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/20' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/12' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/39' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/10' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/34' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/38' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/11' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/16' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/32' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/30' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/14' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/18' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/13' 2024-05-27T06:50:27.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/33' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/17' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/35' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './133/37' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './133' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/19' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/15' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/31' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/36' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/20' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/12' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/39' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/10' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/34' 2024-05-27T06:50:27.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/38' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/11' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/16' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/32' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/30' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/14' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/18' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/13' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/33' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/17' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/35' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './134/37' 2024-05-27T06:50:27.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './134' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/19' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/15' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/31' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/36' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/20' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/12' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/39' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/10' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/34' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/38' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/11' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/16' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/32' 2024-05-27T06:50:27.787 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/30' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/14' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/18' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/13' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/33' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/17' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/35' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './135/37' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './135' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/19' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/15' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/31' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/36' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/20' 2024-05-27T06:50:27.788 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/12' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/39' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/10' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/34' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/38' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/11' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/16' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/32' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/30' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/14' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/18' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/13' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/33' 2024-05-27T06:50:27.789 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/17' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/35' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './136/37' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './136' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/19' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/15' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/31' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/36' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/20' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/12' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/39' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/10' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/34' 2024-05-27T06:50:27.790 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/38' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/11' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/16' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/32' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/30' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/14' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/18' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/13' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/33' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/17' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/35' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './137/37' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './137' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/19' 2024-05-27T06:50:27.791 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/15' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/31' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/36' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/20' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/12' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/39' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/10' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/34' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/38' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/11' 2024-05-27T06:50:27.792 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/16' 2024-05-27T06:50:29.123 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/32' 2024-05-27T06:50:29.123 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/30' 2024-05-27T06:50:29.123 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/14' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/18' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/13' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/33' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/17' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/35' 2024-05-27T06:50:29.124 INFO:tasks.workunit.client.0.smithi081.stdout:removed './138/37' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './138' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/19' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/15' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/31' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/36' 2024-05-27T06:50:29.125 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/20' 2024-05-27T06:50:29.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/12' 2024-05-27T06:50:29.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/39' 2024-05-27T06:50:29.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/10' 2024-05-27T06:50:29.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/34' 2024-05-27T06:50:29.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/38' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/11' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/16' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/32' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/30' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/14' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/18' 2024-05-27T06:50:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/13' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/33' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/17' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/35' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './139/37' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './139' 2024-05-27T06:50:29.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/19' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/15' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/31' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/36' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/20' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/12' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/39' 2024-05-27T06:50:29.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/10' 2024-05-27T06:50:29.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/34' 2024-05-27T06:50:29.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/38' 2024-05-27T06:50:29.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/11' 2024-05-27T06:50:29.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/16' 2024-05-27T06:50:29.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/32' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/30' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/14' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/18' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/13' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/33' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/17' 2024-05-27T06:50:29.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/35' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './140/37' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './140' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/19' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/15' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/31' 2024-05-27T06:50:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/36' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/20' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/12' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/39' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/10' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/34' 2024-05-27T06:50:29.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/38' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/11' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/16' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/32' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/30' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/14' 2024-05-27T06:50:29.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/18' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/13' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/33' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/17' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/35' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './141/37' 2024-05-27T06:50:29.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './141' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/19' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/15' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/31' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/36' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/20' 2024-05-27T06:50:29.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/12' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/39' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/10' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/34' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/38' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/11' 2024-05-27T06:50:29.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/16' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/32' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/30' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/14' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/18' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/13' 2024-05-27T06:50:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/33' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/17' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/35' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './142/37' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './142' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/19' 2024-05-27T06:50:29.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/15' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/31' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/36' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/20' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/12' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/39' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/10' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/34' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/38' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/11' 2024-05-27T06:50:29.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/16' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/32' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/30' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/14' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/18' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/13' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/33' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/17' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/35' 2024-05-27T06:50:29.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './143/37' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './143' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/19' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/15' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/31' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/36' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/20' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/12' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/39' 2024-05-27T06:50:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/10' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/34' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/38' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/11' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/16' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/32' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/30' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/14' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/18' 2024-05-27T06:50:29.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/13' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/33' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/17' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/35' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './144/37' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './144' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/19' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/15' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/31' 2024-05-27T06:50:29.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/36' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/20' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/12' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/39' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/10' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/34' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/38' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/11' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/16' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/32' 2024-05-27T06:50:29.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/30' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/14' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/18' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/13' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/33' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/17' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/35' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './145/37' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './145' 2024-05-27T06:50:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/19' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/15' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/31' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/36' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/20' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/12' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/39' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/10' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/34' 2024-05-27T06:50:29.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/38' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/11' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/16' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/32' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/30' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/14' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/18' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/13' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/33' 2024-05-27T06:50:29.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/17' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/35' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './146/37' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './146' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/19' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/15' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/31' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/36' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/20' 2024-05-27T06:50:29.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/12' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/39' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/10' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/34' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/38' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/11' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/16' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/32' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/30' 2024-05-27T06:50:29.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/14' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/18' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/13' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/33' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/17' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/35' 2024-05-27T06:50:29.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './147/37' 2024-05-27T06:50:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './147' 2024-05-27T06:50:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/19' 2024-05-27T06:50:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/15' 2024-05-27T06:50:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/31' 2024-05-27T06:50:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/36' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/20' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/12' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/39' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/10' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/34' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/38' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/11' 2024-05-27T06:50:30.269 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/16' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/32' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/30' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/14' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/18' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/13' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/33' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/17' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/35' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './148/37' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './148' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/19' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/15' 2024-05-27T06:50:30.270 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/31' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/36' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/20' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/12' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/39' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/10' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/34' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/38' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/11' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/16' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/32' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/30' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/14' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/18' 2024-05-27T06:50:30.271 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/13' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/33' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/17' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/35' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './149/37' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './149' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/19' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/15' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/31' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/36' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/20' 2024-05-27T06:50:30.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/12' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/39' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/10' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/34' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/38' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/11' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/16' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/32' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/30' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/14' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/18' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/13' 2024-05-27T06:50:30.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/33' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/17' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/35' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './150/37' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './150' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/19' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/15' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/31' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/36' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/20' 2024-05-27T06:50:30.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/12' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/39' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/10' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/34' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/38' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/11' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/16' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/32' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/30' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/14' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/18' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/13' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/33' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/17' 2024-05-27T06:50:30.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/35' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './151/37' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './151' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/19' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/15' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/31' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/36' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/20' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/12' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/39' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/10' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/34' 2024-05-27T06:50:30.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/38' 2024-05-27T06:50:30.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/11' 2024-05-27T06:50:30.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/16' 2024-05-27T06:50:30.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/32' 2024-05-27T06:50:30.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/30' 2024-05-27T06:50:30.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/14' 2024-05-27T06:50:30.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/18' 2024-05-27T06:50:30.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/13' 2024-05-27T06:50:30.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/33' 2024-05-27T06:50:30.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/17' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/35' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './152/37' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './152' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/19' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/15' 2024-05-27T06:50:30.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/31' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/36' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/20' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/12' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/39' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/10' 2024-05-27T06:50:30.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/34' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/38' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/11' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/16' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/32' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/30' 2024-05-27T06:50:30.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/14' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/18' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/13' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/33' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/17' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/35' 2024-05-27T06:50:30.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './153/37' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './153' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/19' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/15' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/31' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/36' 2024-05-27T06:50:30.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/20' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/12' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/39' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/10' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/34' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/38' 2024-05-27T06:50:30.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/11' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/16' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/32' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/30' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/14' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/18' 2024-05-27T06:50:30.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/13' 2024-05-27T06:50:30.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/33' 2024-05-27T06:50:30.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/17' 2024-05-27T06:50:30.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/35' 2024-05-27T06:50:30.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './154/37' 2024-05-27T06:50:30.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './154' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/19' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/15' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/31' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/36' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/20' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/12' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/39' 2024-05-27T06:50:30.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/10' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/34' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/38' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/11' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/16' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/32' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/30' 2024-05-27T06:50:30.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/14' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/18' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/13' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/33' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/17' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/35' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './155/37' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './155' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/19' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/15' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/31' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/36' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/20' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/12' 2024-05-27T06:50:30.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/39' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/10' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/34' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/38' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/11' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/16' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/32' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/30' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/14' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/18' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/13' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/33' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/17' 2024-05-27T06:50:30.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/35' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './156/37' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './156' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/19' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/15' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/31' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/36' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/20' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/12' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/39' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/10' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/34' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/38' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/11' 2024-05-27T06:50:30.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/16' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/32' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/30' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/14' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/18' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/13' 2024-05-27T06:50:30.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/33' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/17' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/35' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './157/37' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './157' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/19' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/15' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/31' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/36' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/20' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/12' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/39' 2024-05-27T06:50:31.431 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/10' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/34' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/38' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/11' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/16' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/32' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/30' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/14' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/18' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/13' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/33' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/17' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/35' 2024-05-27T06:50:31.432 INFO:tasks.workunit.client.0.smithi081.stdout:removed './158/37' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './158' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/19' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/15' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/31' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/36' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/20' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/12' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/39' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/10' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/34' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/38' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/11' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/16' 2024-05-27T06:50:31.433 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/32' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/30' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/14' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/18' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/13' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/33' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/17' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/35' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './159/37' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './159' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/19' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/15' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/31' 2024-05-27T06:50:31.434 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/36' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/20' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/12' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/39' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/10' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/34' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/38' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/11' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/16' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/32' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/30' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/14' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/18' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/13' 2024-05-27T06:50:31.435 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/33' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/17' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/35' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './160/37' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './160' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/19' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/15' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/31' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/36' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/20' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/12' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/39' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/10' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/34' 2024-05-27T06:50:31.436 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/38' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/11' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/16' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/32' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/30' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/14' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/18' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/13' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/33' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/17' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/35' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './161/37' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './161' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/19' 2024-05-27T06:50:31.437 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/15' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/31' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/36' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/20' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/12' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/39' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/10' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/34' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/38' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/11' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/16' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/32' 2024-05-27T06:50:31.438 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/30' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/14' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/18' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/13' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/33' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/17' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/35' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './162/37' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './162' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/19' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/15' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/31' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/36' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/20' 2024-05-27T06:50:31.439 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/12' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/39' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/10' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/34' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/38' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/11' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/16' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/32' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/30' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/14' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/18' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/13' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/33' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/17' 2024-05-27T06:50:31.440 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/35' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './163/37' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './163' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/19' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/15' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/31' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/36' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/20' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/12' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/39' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/10' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/34' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/38' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/11' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/16' 2024-05-27T06:50:31.441 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/32' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/30' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/14' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/18' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/13' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/33' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/17' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/35' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './164/37' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './164' 2024-05-27T06:50:31.442 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/19' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/15' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/31' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/36' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/20' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/12' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/39' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/10' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/34' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/38' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/11' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/16' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/32' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/30' 2024-05-27T06:50:31.443 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/14' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/18' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/13' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/33' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/17' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/35' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './165/37' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './165' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/19' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/15' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/31' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/36' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/20' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/12' 2024-05-27T06:50:31.444 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/39' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/10' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/34' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/38' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/11' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/16' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/32' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/30' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/14' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/18' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/13' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/33' 2024-05-27T06:50:31.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/17' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/35' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './166/37' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './166' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/19' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/15' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/31' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/36' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/20' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/12' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/39' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/10' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/34' 2024-05-27T06:50:31.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/38' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/11' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/16' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/32' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/30' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/14' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/18' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/13' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/33' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/17' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/35' 2024-05-27T06:50:32.136 INFO:tasks.workunit.client.0.smithi081.stdout:removed './167/37' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './167' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/19' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/15' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/31' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/36' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/20' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/12' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/39' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/10' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/34' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/38' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/11' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/16' 2024-05-27T06:50:32.137 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/32' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/30' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/14' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/18' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/13' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/33' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/17' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/35' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './168/37' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './168' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/19' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/15' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/31' 2024-05-27T06:50:32.138 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/36' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/20' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/12' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/39' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/10' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/34' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/38' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/11' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/16' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/32' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/30' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/14' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/18' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/13' 2024-05-27T06:50:32.139 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/33' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/17' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/35' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './169/37' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './169' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/19' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/15' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/31' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/36' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/20' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/12' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/39' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/10' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/34' 2024-05-27T06:50:32.140 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/38' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/11' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/16' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/32' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/30' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/14' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/18' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/13' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/33' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/17' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/35' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './170/37' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './170' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/19' 2024-05-27T06:50:32.141 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/15' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/31' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/36' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/20' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/12' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/39' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/10' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/34' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/38' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/11' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/16' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/32' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/30' 2024-05-27T06:50:32.142 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/14' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/18' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/13' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/33' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/17' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/35' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './171/37' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './171' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/19' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/15' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/31' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/36' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/20' 2024-05-27T06:50:32.143 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/12' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/39' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/10' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/34' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/38' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/11' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/16' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/32' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/30' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/14' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/18' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/13' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/33' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/17' 2024-05-27T06:50:32.144 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/35' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './172/37' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './172' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/19' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/15' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/31' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/36' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/20' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/12' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/39' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/10' 2024-05-27T06:50:32.145 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/34' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/38' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/11' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/16' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/32' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/30' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/14' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/18' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/13' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/33' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/17' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/35' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed './173/37' 2024-05-27T06:50:32.146 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './173' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/19' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/15' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/31' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/36' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/20' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/12' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/39' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/10' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/34' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/38' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/11' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/16' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/32' 2024-05-27T06:50:32.147 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/30' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/14' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/18' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/13' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/33' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/17' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/35' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './174/37' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './174' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/19' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/15' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/31' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/36' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/20' 2024-05-27T06:50:32.148 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/12' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/39' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/10' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/34' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/38' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/11' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/16' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/32' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/30' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/14' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/18' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/13' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/33' 2024-05-27T06:50:32.149 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/17' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/35' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './175/37' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './175' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/19' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/15' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/31' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/36' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/20' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/12' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/39' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/10' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/34' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/38' 2024-05-27T06:50:32.150 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/11' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/16' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/32' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/30' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/14' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/18' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/13' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/33' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/17' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/35' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './176/37' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './176' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/19' 2024-05-27T06:50:32.151 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/15' 2024-05-27T06:50:32.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:50:32.170 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/31' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/36' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/20' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/12' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/39' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/10' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/34' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/38' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/11' 2024-05-27T06:50:32.829 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/16' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/32' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/30' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/14' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/18' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/13' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/33' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/17' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/35' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './177/37' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './177' 2024-05-27T06:50:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/19' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/15' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/31' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/36' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/20' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/12' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/39' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/10' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/34' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/38' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/11' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/16' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/32' 2024-05-27T06:50:32.831 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/30' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/14' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/18' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/13' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/33' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/17' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/35' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './178/37' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './178' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/19' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/15' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/31' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/36' 2024-05-27T06:50:32.832 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/20' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/12' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/39' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/10' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/34' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/38' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/11' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/16' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/32' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/30' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/14' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/18' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/13' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/33' 2024-05-27T06:50:32.833 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/17' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/35' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './179/37' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './179' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/19' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/15' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/31' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/36' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/20' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/12' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/39' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/10' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/34' 2024-05-27T06:50:32.834 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/38' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/11' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/16' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/32' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/30' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/14' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/18' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/13' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/33' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/17' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/35' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './180/37' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './180' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/19' 2024-05-27T06:50:32.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/15' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/31' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/36' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/20' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/12' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/39' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/10' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/34' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/38' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/11' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/16' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/32' 2024-05-27T06:50:32.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/30' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/14' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/18' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/13' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/33' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/17' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/35' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './181/37' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './181' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/19' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/15' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/31' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/36' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/20' 2024-05-27T06:50:32.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/12' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/39' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/10' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/34' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/38' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/11' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/16' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/32' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/30' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/14' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/18' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/13' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/33' 2024-05-27T06:50:32.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/17' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/35' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './182/37' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './182' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/19' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/15' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/31' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/36' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/20' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/12' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/39' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/10' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/34' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/38' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/11' 2024-05-27T06:50:32.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/16' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/32' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/30' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/14' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/18' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/13' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/33' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/17' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/35' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './183/37' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './183' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/19' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/15' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/31' 2024-05-27T06:50:32.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/36' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/20' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/12' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/39' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/10' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/34' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/38' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/11' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/16' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/32' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/30' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/14' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/18' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/13' 2024-05-27T06:50:32.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/33' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/17' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/35' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './184/37' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './184' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/19' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/15' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/31' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/36' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/20' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/12' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/39' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/10' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/34' 2024-05-27T06:50:32.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/38' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/11' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/16' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/32' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/30' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/14' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/18' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/13' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/33' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/17' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/35' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './185/37' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './185' 2024-05-27T06:50:32.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/19' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/15' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/31' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/36' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/20' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/12' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/39' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/10' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/34' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/38' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/11' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/16' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/32' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/30' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/14' 2024-05-27T06:50:32.844 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/18' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/13' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/33' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/17' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/35' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './186/37' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './186' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/19' 2024-05-27T06:50:34.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/15' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/31' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/36' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/20' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/12' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/39' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/10' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/34' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/38' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/11' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/16' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/32' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/30' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/14' 2024-05-27T06:50:34.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/18' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/13' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/33' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/17' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/35' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './187/37' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './187' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/19' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/15' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/31' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/36' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/20' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/12' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/39' 2024-05-27T06:50:34.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/10' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/34' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/38' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/11' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/16' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/32' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/30' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/14' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/18' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/13' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/33' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/17' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/35' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './188/37' 2024-05-27T06:50:34.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './188' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/19' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/15' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/31' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/36' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/20' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/12' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/39' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/10' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/34' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/38' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/11' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/16' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/32' 2024-05-27T06:50:34.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/30' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/14' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/18' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/13' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/33' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/17' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/35' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './189/37' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './189' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/19' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/15' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/31' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/36' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/20' 2024-05-27T06:50:34.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/12' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/39' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/10' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/34' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/38' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/11' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/16' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/32' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/30' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/14' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/18' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/13' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/33' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/17' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/35' 2024-05-27T06:50:34.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './190/37' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './190' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/19' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/15' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/31' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/36' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/20' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/12' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/39' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/10' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/34' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/38' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/11' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/16' 2024-05-27T06:50:34.629 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/32' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/30' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/14' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/18' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/13' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/33' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/17' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/35' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './191/37' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './191' 2024-05-27T06:50:34.630 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/19' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/15' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/31' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/36' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/20' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/12' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/39' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/10' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/34' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/38' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/11' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/16' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/32' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/30' 2024-05-27T06:50:34.631 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/14' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/18' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/13' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/33' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/17' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/35' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './192/37' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './192' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/19' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/15' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/31' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/36' 2024-05-27T06:50:34.632 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/20' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/12' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/39' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/10' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/34' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/38' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/11' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/16' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/32' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/30' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/14' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/18' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/13' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/33' 2024-05-27T06:50:34.633 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/17' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/35' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './193/37' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './193' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/19' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/15' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/31' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/36' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/20' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/12' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/39' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/10' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/34' 2024-05-27T06:50:34.634 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/38' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/11' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/16' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/32' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/30' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/14' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/18' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/13' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/33' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/17' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/35' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './194/37' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './194' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/19' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/15' 2024-05-27T06:50:34.635 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/31' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/36' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/20' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/12' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/39' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/10' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/34' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/38' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/11' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/16' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/32' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/30' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/14' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/18' 2024-05-27T06:50:34.636 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/13' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/33' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/17' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/35' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './195/37' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './195' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/19' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/15' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/31' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/36' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/20' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/12' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/39' 2024-05-27T06:50:34.637 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/10' 2024-05-27T06:50:34.896 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:35.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.076+0000 7f0675000640 1 Processor -- start 2024-05-27T06:50:35.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.076+0000 7f0675000640 1 -- start start 2024-05-27T06:50:35.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f0675000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f0675000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f067015a040 0x7f06700c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f0675000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0670072840 con 0x7f06700c60a0 2024-05-27T06:50:35.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f0675000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f06700c5040 con 0x7f067015a040 2024-05-27T06:50:35.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f0675000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f06700c75c0 con 0x7f0670072430 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066e800640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f06700c60a0 0x7f067015c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:60886/0 (socket says 172.21.15.81:60886) 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066e800640 1 -- 172.21.15.81:0/1485589320 learned_addr learned my addr 172.21.15.81:0/1485589320 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066d400640 1 --2- 172.21.15.81:0/1485589320 >> v2:172.21.15.106:3300/0 conn(0x7f067015a040 0x7f06700c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066de00640 1 --2- 172.21.15.81:0/1485589320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066ca00640 1 -- 172.21.15.81:0/1485589320 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525102583 0 0) 0x7f0670072840 con 0x7f06700c60a0 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066ca00640 1 -- 172.21.15.81:0/1485589320 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0654003530 con 0x7f06700c60a0 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066de00640 1 -- 172.21.15.81:0/1485589320 >> v2:172.21.15.106:3300/0 conn(0x7f067015a040 msgr2=0x7f06700c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066de00640 1 --2- 172.21.15.81:0/1485589320 >> v2:172.21.15.106:3300/0 conn(0x7f067015a040 0x7f06700c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066de00640 1 -- 172.21.15.81:0/1485589320 >> v1:172.21.15.106:6789/0 conn(0x7f06700c60a0 legacy=0x7f067015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.080+0000 7f066de00640 1 -- 172.21.15.81:0/1485589320 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06700c5270 con 0x7f0670072430 2024-05-27T06:50:35.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.084+0000 7f066de00640 1 --2- 172.21.15.81:0/1485589320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670159b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f066000dbe0 tx=0x7f06600207b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad11e6fa3093b42 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.084+0000 7f066ca00640 1 -- 172.21.15.81:0/1485589320 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600b0070 con 0x7f0670072430 2024-05-27T06:50:35.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.084+0000 7f066ca00640 1 -- 172.21.15.81:0/1485589320 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066000b450 con 0x7f0670072430 2024-05-27T06:50:35.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.084+0000 7f066ca00640 1 -- 172.21.15.81:0/1485589320 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066000d2d0 con 0x7f0670072430 2024-05-27T06:50:35.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- 172.21.15.81:0/1485589320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 msgr2=0x7f0670159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 --2- 172.21.15.81:0/1485589320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670159b00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f066000dbe0 tx=0x7f06600207b0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- 172.21.15.81:0/1485589320 shutdown_connections 2024-05-27T06:50:35.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 --2- 172.21.15.81:0/1485589320 >> v2:172.21.15.106:3300/0 conn(0x7f067015a040 0x7f06700c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 --2- 172.21.15.81:0/1485589320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670159b00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- 172.21.15.81:0/1485589320 >> 172.21.15.81:0/1485589320 conn(0x7f0670070700 msgr2=0x7f06700cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:35.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- 172.21.15.81:0/1485589320 shutdown_connections 2024-05-27T06:50:35.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- 172.21.15.81:0/1485589320 wait complete. 2024-05-27T06:50:35.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 Processor -- start 2024-05-27T06:50:35.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.088+0000 7f0675000640 1 -- start start 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 0x7f06701581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06700ce6f0 con 0x7f067015a040 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f06700c7dd0 con 0x7f06700c60a0 2024-05-27T06:50:35.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f06700c5610 con 0x7f0670072430 2024-05-27T06:50:35.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57868/0 (socket says 172.21.15.81:57868) 2024-05-27T06:50:35.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 -- 172.21.15.81:0/3266933282 learned_addr learned my addr 172.21.15.81:0/3266933282 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066de00640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066d400640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 0x7f06701581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 -- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 msgr2=0x7f0670157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 -- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 msgr2=0x7f06701581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 0x7f06701581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 -- 172.21.15.81:0/3266933282 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0670157960 con 0x7f067015a040 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066d400640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 0x7f06701581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f066e800640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f066800caa0 tx=0x7f066800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066800ac00 con 0x7f067015a040 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066800ada0 con 0x7f067015a040 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668012600 con 0x7f067015a040 2024-05-27T06:50:35.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067015afb0 con 0x7f067015a040 2024-05-27T06:50:35.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067015b250 con 0x7f067015a040 2024-05-27T06:50:35.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067006a560 con 0x7f067015a040 2024-05-27T06:50:35.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.092+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f06680127a0 con 0x7f067015a040 2024-05-27T06:50:35.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.096+0000 7f0666000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 0x7f0650071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.096+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7f066808fa80 con 0x7f067015a040 2024-05-27T06:50:35.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.096+0000 7f066de00640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 0x7f0650071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.096+0000 7f066de00640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 0x7f0650071940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f06600b2000 tx=0x7f06600b8040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.100+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f066805bff0 con 0x7f067015a040 2024-05-27T06:50:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f067015ba50 con 0x7f067015a040 2024-05-27T06:50:35.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0666000640 1 -- 172.21.15.81:0/3266933282 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7f06680602f0 con 0x7f067015a040 2024-05-27T06:50:35.288 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:35.290 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:35.293 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:35.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 msgr2=0x7f0650071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 0x7f0650071940 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f06600b2000 tx=0x7f06600b8040 comp rx=0 tx=0).stop 2024-05-27T06:50:35.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 msgr2=0x7f0670158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f066800caa0 tx=0x7f066800cf60 comp rx=0 tx=0).stop 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 shutdown_connections 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f065006f430 0x7f0650071940 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f067015a040 0x7f0670158730 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06700c60a0 0x7f06701581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 --2- 172.21.15.81:0/3266933282 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0670072430 0x7f0670157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 >> 172.21.15.81:0/3266933282 conn(0x7f0670070700 msgr2=0x7f0670074830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 shutdown_connections 2024-05-27T06:50:35.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.292+0000 7f0675000640 1 -- 172.21.15.81:0/3266933282 wait complete. 2024-05-27T06:50:35.309 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=3) 2024-05-27T06:50:35.309 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T06:50:35.309 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:50:35.324 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T06:50:35.325 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:35.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.520+0000 7fb9df200640 1 Processor -- start 2024-05-27T06:50:35.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.520+0000 7fb9df200640 1 -- start start 2024-05-27T06:50:35.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9df200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb9e0075200 0x7fb9e0168180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9df200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e00c9df0 con 0x7fb9e00ca1e0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9df200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e015d7e0 con 0x7fb9e0156cc0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9df200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb9e0074660 con 0x7fb9e0075200 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dde00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb9e00ca1e0 0x7fb9e0071f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39582/0 (socket says 172.21.15.81:39582) 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dde00640 1 -- 172.21.15.81:0/3959903740 learned_addr learned my addr 172.21.15.81:0/3959903740 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9de800640 1 --2- 172.21.15.81:0/3959903740 >> v2:172.21.15.106:3300/0 conn(0x7fb9e0075200 0x7fb9e0168180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389410965 0 0) 0x7fb9d00012a0 con 0x7fb9e00ca1e0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9c4003530 con 0x7fb9e00ca1e0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167154686 0 0) 0x7fb9e015d7e0 con 0x7fb9e0156cc0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d00012a0 con 0x7fb9e0156cc0 2024-05-27T06:50:35.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9de800640 1 -- 172.21.15.81:0/3959903740 >> v1:172.21.15.106:6789/0 conn(0x7fb9e0156cc0 legacy=0x7fb9e0074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9de800640 1 -- 172.21.15.81:0/3959903740 >> v1:172.21.15.81:6789/0 conn(0x7fb9e00ca1e0 legacy=0x7fb9e0071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.524+0000 7fb9de800640 1 -- 172.21.15.81:0/3959903740 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e015cc60 con 0x7fb9e0075200 2024-05-27T06:50:35.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9de800640 1 --2- 172.21.15.81:0/3959903740 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e0075200 0x7fb9e0168180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009d90 tx=0x7fb9d801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07a0121f1dc82a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80b2070 con 0x7fb9e0075200 2024-05-27T06:50:35.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d80a5890 con 0x7fb9e0075200 2024-05-27T06:50:35.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9dca00640 1 -- 172.21.15.81:0/3959903740 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80a5c40 con 0x7fb9e0075200 2024-05-27T06:50:35.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 -- 172.21.15.81:0/3959903740 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e0075200 msgr2=0x7fb9e0168180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 --2- 172.21.15.81:0/3959903740 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e0075200 0x7fb9e0168180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009d90 tx=0x7fb9d801c8e0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 -- 172.21.15.81:0/3959903740 shutdown_connections 2024-05-27T06:50:35.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 --2- 172.21.15.81:0/3959903740 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e0075200 0x7fb9e0168180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 -- 172.21.15.81:0/3959903740 >> 172.21.15.81:0/3959903740 conn(0x7fb9e0068cb0 msgr2=0x7fb9e00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:35.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 -- 172.21.15.81:0/3959903740 shutdown_connections 2024-05-27T06:50:35.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.528+0000 7fb9df200640 1 -- 172.21.15.81:0/3959903740 wait complete. 2024-05-27T06:50:35.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 Processor -- start 2024-05-27T06:50:35.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- start start 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e00ca1e0 0x7fb9e00c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9e0156cc0 0x7fb9e015fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e015ca10 con 0x7fb9e0075200 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e015c890 con 0x7fb9e00ca1e0 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e00726a0 con 0x7fb9e0156cc0 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dd400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e00ca1e0 0x7fb9e00c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34560/0 (socket says 172.21.15.81:34560) 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 -- 172.21.15.81:0/1818591960 learned_addr learned my addr 172.21.15.81:0/1818591960 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9de800640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9e0156cc0 0x7fb9e015fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 -- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9e0156cc0 msgr2=0x7fb9e015fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9e0156cc0 0x7fb9e015fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 -- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e00ca1e0 msgr2=0x7fb9e00c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e00ca1e0 0x7fb9e00c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 -- 172.21.15.81:0/1818591960 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e00c7830 con 0x7fb9e0075200 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9dde00640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000c730 tx=0x7fb9d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0007270 con 0x7fb9e0075200 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d0007410 con 0x7fb9e0075200 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e015f770 con 0x7fb9e0075200 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00059b0 con 0x7fb9e0075200 2024-05-27T06:50:35.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.532+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e01f62a0 con 0x7fb9e0075200 2024-05-27T06:50:35.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e0050870 con 0x7fb9e0075200 2024-05-27T06:50:35.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb9d0005b50 con 0x7fb9e0075200 2024-05-27T06:50:35.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9d6000640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 0x7fb9c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7fb9d008fdf0 con 0x7fb9e0075200 2024-05-27T06:50:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9dd400640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 0x7fb9c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:35.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.536+0000 7fb9dd400640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 0x7fb9c0071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c10 tx=0x7fb9cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:35.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.540+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb9e00c9df0 con 0x7fb9e0075200 2024-05-27T06:50:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.756+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9e00cab90 con 0x7fb9e0075200 2024-05-27T06:50:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.756+0000 7fb9d6000640 1 -- 172.21.15.81:0/1818591960 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7fb9d005c2b0 con 0x7fb9e0075200 2024-05-27T06:50:35.752 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:35.752 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:35.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 msgr2=0x7fb9c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 0x7fb9c0071910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c10 tx=0x7fb9cc009040 comp rx=0 tx=0).stop 2024-05-27T06:50:35.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 msgr2=0x7fb9e00c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:35.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb9d000c730 tx=0x7fb9d000cbf0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 shutdown_connections 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb9e0156cc0 0x7fb9e015fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb9c006f400 0x7fb9c0071910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb9e00ca1e0 0x7fb9e00c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 --2- 172.21.15.81:0/1818591960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9e0075200 0x7fb9e00c72f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 >> 172.21.15.81:0/1818591960 conn(0x7fb9e0068cb0 msgr2=0x7fb9e015bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 shutdown_connections 2024-05-27T06:50:35.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:35.760+0000 7fb9df200640 1 -- 172.21.15.81:0/1818591960 wait complete. 2024-05-27T06:50:35.775 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/34' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/38' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/11' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/16' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/32' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/30' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/14' 2024-05-27T06:50:36.187 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/18' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/13' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/33' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/17' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/35' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './196/37' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './196' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/19' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/15' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/31' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/36' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/20' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/12' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/39' 2024-05-27T06:50:36.188 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/10' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/34' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/38' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/11' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/16' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/32' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/30' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/14' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/18' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/13' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/33' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/17' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/35' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed './197/37' 2024-05-27T06:50:36.189 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './197' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/19' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/15' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/31' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/36' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/20' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/12' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/39' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/10' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/34' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/38' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/11' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/16' 2024-05-27T06:50:36.190 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/32' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/30' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/14' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/18' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/13' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/33' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/17' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/35' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './198/37' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './198' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/19' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/15' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/31' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/36' 2024-05-27T06:50:36.191 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/20' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/12' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/39' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/10' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/34' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/38' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/11' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/16' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/32' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/30' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/14' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/18' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/13' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/33' 2024-05-27T06:50:36.192 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/17' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/35' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './199/37' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './199' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/21' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/27' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/31' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/36' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/20' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/39' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/34' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/38' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/28' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/24' 2024-05-27T06:50:36.193 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/29' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/32' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/22' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/25' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/30' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/33' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/23' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/35' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/26' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './200/37' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './200' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/21' 2024-05-27T06:50:36.194 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/27' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/31' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/36' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/20' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/39' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/34' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/38' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/28' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/24' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/29' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/32' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/22' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/25' 2024-05-27T06:50:36.195 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/30' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/33' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/23' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/35' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/26' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './201/37' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './201' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/21' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/27' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/31' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/36' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/20' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/39' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/34' 2024-05-27T06:50:36.196 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/38' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/28' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/24' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/29' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/32' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/22' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/25' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/30' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/33' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/23' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/35' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/26' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './202/37' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './202' 2024-05-27T06:50:36.197 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/21' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/27' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/31' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/36' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/20' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/39' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/34' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/38' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/28' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/24' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/29' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/32' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/22' 2024-05-27T06:50:36.198 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/25' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/30' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/33' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/23' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/35' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/26' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './203/37' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './203' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/21' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/27' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/31' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/36' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/20' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/39' 2024-05-27T06:50:36.199 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/34' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/38' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/28' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/24' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/29' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/32' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/22' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/25' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/30' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/33' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/23' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/35' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/26' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './204/37' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './204' 2024-05-27T06:50:36.200 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/21' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/27' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/31' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/36' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/20' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/39' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/34' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/38' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/28' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/24' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/29' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/32' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/22' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/25' 2024-05-27T06:50:36.201 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/30' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/33' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/23' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/35' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/26' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './205/37' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './205' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/21' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/27' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/31' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/36' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/20' 2024-05-27T06:50:36.202 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/39' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/34' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/38' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/28' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/24' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/29' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/32' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/22' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/25' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/30' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/33' 2024-05-27T06:50:36.797 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/23' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/35' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/26' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './206/37' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './206' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/21' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/27' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/31' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/36' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/20' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/39' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/34' 2024-05-27T06:50:36.798 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/38' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/28' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/24' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/29' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/32' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/22' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/25' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/30' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/33' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/23' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/35' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/26' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './207/37' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './207' 2024-05-27T06:50:36.799 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/21' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/27' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/31' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/36' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/20' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/39' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/34' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/38' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/28' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/24' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/29' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/32' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/22' 2024-05-27T06:50:36.800 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/25' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/30' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/33' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/23' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/35' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/26' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './208/37' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './208' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/21' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/27' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/31' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/36' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/20' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/39' 2024-05-27T06:50:36.801 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/34' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/38' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/28' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/24' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/29' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/32' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/22' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/25' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/30' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/33' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/23' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/35' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/26' 2024-05-27T06:50:36.802 INFO:tasks.workunit.client.0.smithi081.stdout:removed './209/37' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './209' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/21' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/27' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/31' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/36' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/20' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/39' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/34' 2024-05-27T06:50:36.803 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/38' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/28' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/24' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/29' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/32' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/22' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/25' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/30' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/33' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/23' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/35' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/26' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed './210/37' 2024-05-27T06:50:36.804 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './210' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/21' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/27' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/31' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/36' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/20' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/39' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/34' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/38' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/28' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/24' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/29' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/32' 2024-05-27T06:50:36.805 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/22' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/25' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/30' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/33' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/23' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/35' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/26' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './211/37' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './211' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/21' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/27' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/31' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/36' 2024-05-27T06:50:36.806 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/20' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/39' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/34' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/38' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/28' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/24' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/29' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/32' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/22' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/25' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/30' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/33' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/23' 2024-05-27T06:50:36.807 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/35' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/26' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './212/37' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './212' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/21' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/27' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/31' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/36' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/20' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/39' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/34' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/38' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/28' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/24' 2024-05-27T06:50:36.808 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/29' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/32' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/22' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/25' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/30' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/33' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/23' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/35' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/26' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './213/37' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './213' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/21' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/27' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/31' 2024-05-27T06:50:36.809 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/36' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/20' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/39' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/34' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/38' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/28' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/24' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/29' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/32' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/22' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/25' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/30' 2024-05-27T06:50:36.810 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/33' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/23' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/35' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/26' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './214/37' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './214' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/21' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/27' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/31' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/36' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/20' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/39' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/34' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/38' 2024-05-27T06:50:36.811 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/28' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/24' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/29' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/32' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/22' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/25' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/30' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/33' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/23' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/35' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/26' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './215/37' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './215' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/21' 2024-05-27T06:50:36.812 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/27' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/31' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/36' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/20' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/39' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/34' 2024-05-27T06:50:36.813 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/38' 2024-05-27T06:50:37.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:37.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.932+0000 7f0a91c00640 1 Processor -- start 2024-05-27T06:50:37.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.932+0000 7f0a91c00640 1 -- start start 2024-05-27T06:50:37.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 0x7f0a8c072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a8c072a60 0x7f0a8c164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a8c0c9c70 con 0x7f0a8c0c9630 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c0c9a40 con 0x7f0a8c072a60 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c15b0c0 con 0x7f0a8c154780 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a8b400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 0x7f0a8c072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a8c072a60 0x7f0a8c164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a8be00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f0a8c0c9630 0x7f0a8c15dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:60894/0 (socket says 172.21.15.81:60894) 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a8be00640 1 -- 172.21.15.81:0/2205256901 learned_addr learned my addr 172.21.15.81:0/2205256901 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a8b400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 0x7f0a8c072520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:42842/0 (socket says 172.21.15.81:42842) 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91200640 1 -- 172.21.15.81:0/2205256901 >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 msgr2=0x7f0a8c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91200640 1 --2- 172.21.15.81:0/2205256901 >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 0x7f0a8c072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91200640 1 -- 172.21.15.81:0/2205256901 >> v1:172.21.15.106:6789/0 conn(0x7f0a8c0c9630 legacy=0x7f0a8c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a91200640 1 -- 172.21.15.81:0/2205256901 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c15a4d0 con 0x7f0a8c072a60 2024-05-27T06:50:37.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.936+0000 7f0a8aa00640 1 -- 172.21.15.81:0/2205256901 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 774219252 0 0) 0x7f0a8c0c9c70 con 0x7f0a8c0c9630 2024-05-27T06:50:37.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91200640 1 --2- 172.21.15.81:0/2205256901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c164e50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c9550 tx=0x7f0a8401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0367e02b27bf16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:37.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a8aa00640 1 -- 172.21.15.81:0/2205256901 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840b2070 con 0x7f0a8c072a60 2024-05-27T06:50:37.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a8aa00640 1 -- 172.21.15.81:0/2205256901 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a840a5870 con 0x7f0a8c072a60 2024-05-27T06:50:37.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a8aa00640 1 -- 172.21.15.81:0/2205256901 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a840a5c40 con 0x7f0a8c072a60 2024-05-27T06:50:37.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 -- 172.21.15.81:0/2205256901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 msgr2=0x7f0a8c164e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 --2- 172.21.15.81:0/2205256901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c164e50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c9550 tx=0x7f0a8401c8a0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 -- 172.21.15.81:0/2205256901 shutdown_connections 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 --2- 172.21.15.81:0/2205256901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c164e50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 --2- 172.21.15.81:0/2205256901 >> v2:172.21.15.106:3300/0 conn(0x7f0a8c154780 0x7f0a8c072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 -- 172.21.15.81:0/2205256901 >> 172.21.15.81:0/2205256901 conn(0x7f0a8c068880 msgr2=0x7f0a8c068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 -- 172.21.15.81:0/2205256901 shutdown_connections 2024-05-27T06:50:37.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.940+0000 7f0a91c00640 1 -- 172.21.15.81:0/2205256901 wait complete. 2024-05-27T06:50:37.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 Processor -- start 2024-05-27T06:50:37.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 -- start start 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a8c154780 0x7f0a8c151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c0cd280 con 0x7f0a8c072a60 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c15ad50 con 0x7f0a8c0c9630 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c15e1d0 con 0x7f0a8c154780 2024-05-27T06:50:37.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:42844/0 (socket says 172.21.15.81:42844) 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 -- 172.21.15.81:0/430615349 learned_addr learned my addr 172.21.15.81:0/430615349 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a91200640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a8c154780 0x7f0a8c151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 -- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a8c154780 msgr2=0x7f0a8c151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a8c154780 0x7f0a8c151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 -- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 msgr2=0x7f0a8c156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:37.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 -- 172.21.15.81:0/430615349 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c072890 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.944+0000 7f0a8b400640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800e870 tx=0x7f0a7800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78017070 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a7800ee60 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7800cb40 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c156cb0 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c160bd0 con 0x7f0a8c0c9630 2024-05-27T06:50:37.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c050870 con 0x7f0a8c0c9630 2024-05-27T06:50:37.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0a7801a070 con 0x7f0a8c0c9630 2024-05-27T06:50:37.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 0x7f0a6c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:37.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7f0a7808f740 con 0x7f0a8c0c9630 2024-05-27T06:50:37.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.948+0000 7f0a8be00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 0x7f0a6c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:37.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.952+0000 7f0a8be00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 0x7f0a6c071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c007c40 tx=0x7f0a7c007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:37.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:37.956+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0a7805bca0 con 0x7f0a8c0c9630 2024-05-27T06:50:38.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a8c0cabe0 con 0x7f0a8c0c9630 2024-05-27T06:50:38.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a83e00640 1 -- 172.21.15.81:0/430615349 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7f0a7805ffa0 con 0x7f0a8c0c9630 2024-05-27T06:50:38.187 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:38.187 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:38.188 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 msgr2=0x7f0a6c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:38.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 0x7f0a6c071940 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c007c40 tx=0x7f0a7c007b90 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 msgr2=0x7f0a8c1514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800e870 tx=0x7f0a7800ed30 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 shutdown_connections 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0a8c154780 0x7f0a8c151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0a6c06f430 0x7f0a6c071940 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a8c0c9630 0x7f0a8c1514f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 --2- 172.21.15.81:0/430615349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c072a60 0x7f0a8c156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 >> 172.21.15.81:0/430615349 conn(0x7f0a8c068880 msgr2=0x7f0a8c15d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 shutdown_connections 2024-05-27T06:50:38.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:38.192+0000 7f0a91c00640 1 -- 172.21.15.81:0/430615349 wait complete. 2024-05-27T06:50:38.209 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:39.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/28' 2024-05-27T06:50:39.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/24' 2024-05-27T06:50:39.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/29' 2024-05-27T06:50:39.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/32' 2024-05-27T06:50:39.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/22' 2024-05-27T06:50:39.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/25' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/30' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/33' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/23' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/35' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/26' 2024-05-27T06:50:39.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './216/37' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './216' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/21' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/27' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/31' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/36' 2024-05-27T06:50:39.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/20' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/39' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/34' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/38' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/28' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/24' 2024-05-27T06:50:39.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/29' 2024-05-27T06:50:39.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/32' 2024-05-27T06:50:39.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/22' 2024-05-27T06:50:39.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/25' 2024-05-27T06:50:39.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/30' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/33' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/23' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/35' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/26' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './217/37' 2024-05-27T06:50:39.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './217' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/21' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/27' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/31' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/36' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/20' 2024-05-27T06:50:39.282 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/39' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/34' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/38' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/28' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/24' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/29' 2024-05-27T06:50:39.283 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/32' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/22' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/25' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/30' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/33' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/23' 2024-05-27T06:50:39.284 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/35' 2024-05-27T06:50:39.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/26' 2024-05-27T06:50:39.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './218/37' 2024-05-27T06:50:39.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './218' 2024-05-27T06:50:39.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/21' 2024-05-27T06:50:39.285 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/27' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/31' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/36' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/20' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/39' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/34' 2024-05-27T06:50:39.286 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/38' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/28' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/24' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/29' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/32' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/22' 2024-05-27T06:50:39.287 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/25' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/30' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/33' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/23' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/35' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/26' 2024-05-27T06:50:39.288 INFO:tasks.workunit.client.0.smithi081.stdout:removed './219/37' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './219' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/21' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/27' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/31' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/36' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/20' 2024-05-27T06:50:39.289 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/39' 2024-05-27T06:50:39.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/34' 2024-05-27T06:50:39.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/38' 2024-05-27T06:50:39.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/28' 2024-05-27T06:50:39.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/24' 2024-05-27T06:50:39.290 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/29' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/32' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/22' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/25' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/30' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/33' 2024-05-27T06:50:39.291 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/23' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/35' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/26' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './220/37' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './220' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/21' 2024-05-27T06:50:39.292 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/27' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/31' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/36' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/20' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/39' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/34' 2024-05-27T06:50:39.293 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/38' 2024-05-27T06:50:39.294 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/28' 2024-05-27T06:50:39.294 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/24' 2024-05-27T06:50:39.294 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/29' 2024-05-27T06:50:39.294 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/32' 2024-05-27T06:50:39.294 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/22' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/25' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/30' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/33' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/23' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/35' 2024-05-27T06:50:39.295 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/26' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed './221/37' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './221' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/21' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/27' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/31' 2024-05-27T06:50:39.296 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/36' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/20' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/39' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/34' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/38' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/28' 2024-05-27T06:50:39.297 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/24' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/29' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/32' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/22' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/25' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/30' 2024-05-27T06:50:39.298 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/33' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/23' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/35' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/26' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed './222/37' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './222' 2024-05-27T06:50:39.299 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/21' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/27' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/31' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/36' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/20' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/39' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/34' 2024-05-27T06:50:39.300 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/38' 2024-05-27T06:50:39.301 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/28' 2024-05-27T06:50:39.301 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/24' 2024-05-27T06:50:39.301 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/29' 2024-05-27T06:50:39.301 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/32' 2024-05-27T06:50:39.301 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/22' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/25' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/30' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/33' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/23' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/35' 2024-05-27T06:50:39.302 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/26' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed './223/37' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './223' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/21' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/27' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/31' 2024-05-27T06:50:39.303 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/36' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/20' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/39' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/34' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/38' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/28' 2024-05-27T06:50:39.304 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/24' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/29' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/32' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/22' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/25' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/30' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/33' 2024-05-27T06:50:39.305 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/23' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/35' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/26' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './224/37' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './224' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/21' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/27' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/31' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/36' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/20' 2024-05-27T06:50:39.306 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/39' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/34' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/38' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/28' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/24' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/29' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/32' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/22' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/25' 2024-05-27T06:50:39.307 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/30' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/33' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/23' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/35' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/26' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './225/37' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './225' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/21' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/27' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/31' 2024-05-27T06:50:39.308 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/36' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/20' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/39' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/34' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/38' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/28' 2024-05-27T06:50:39.309 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/24' 2024-05-27T06:50:40.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:40.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.360+0000 7f1349400640 1 Processor -- start 2024-05-27T06:50:40.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.360+0000 7f1349400640 1 -- start start 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1349400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13440c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1349400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1344161cd0 0x7f13441640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1349400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1344072840 con 0x7f1344072430 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1349400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f13440c5040 con 0x7f1344161cd0 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1349400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f13440c75c0 con 0x7f13440c5b60 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1342800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1344072430 0x7f1344159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39596/0 (socket says 172.21.15.81:39596) 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1342800640 1 -- 172.21.15.81:0/1237091224 learned_addr learned my addr 172.21.15.81:0/1237091224 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1343200640 1 --2- 172.21.15.81:0/1237091224 >> v2:172.21.15.81:3300/0 conn(0x7f1344161cd0 0x7f13441640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1341e00640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13440c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1341400640 1 -- 172.21.15.81:0/1237091224 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125441332 0 0) 0x7f1344072840 con 0x7f1344072430 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1341400640 1 -- 172.21.15.81:0/1237091224 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1328003530 con 0x7f1344072430 2024-05-27T06:50:40.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1343200640 1 -- 172.21.15.81:0/1237091224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 msgr2=0x7f13440c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:40.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1343200640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13440c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1343200640 1 -- 172.21.15.81:0/1237091224 >> v1:172.21.15.81:6789/0 conn(0x7f1344072430 legacy=0x7f1344159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1343200640 1 -- 172.21.15.81:0/1237091224 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13440c5270 con 0x7f1344161cd0 2024-05-27T06:50:40.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.364+0000 7f1341e00640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13440c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:50:40.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1343200640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441640f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f133c009d90 tx=0x7f133c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3f5d1674f5dae5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:40.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1341400640 1 -- 172.21.15.81:0/1237091224 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0b2070 con 0x7f1344161cd0 2024-05-27T06:50:40.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1341400640 1 -- 172.21.15.81:0/1237091224 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f133c0a5830 con 0x7f1344161cd0 2024-05-27T06:50:40.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1341400640 1 -- 172.21.15.81:0/1237091224 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0a5be0 con 0x7f1344161cd0 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 -- 172.21.15.81:0/1237091224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 msgr2=0x7f13441640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441640f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f133c009d90 tx=0x7f133c01c970 comp rx=0 tx=0).stop 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 -- 172.21.15.81:0/1237091224 shutdown_connections 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441640f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 --2- 172.21.15.81:0/1237091224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13440c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 -- 172.21.15.81:0/1237091224 >> 172.21.15.81:0/1237091224 conn(0x7f1344070700 msgr2=0x7f13440cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 -- 172.21.15.81:0/1237091224 shutdown_connections 2024-05-27T06:50:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.368+0000 7f1349400640 1 -- 172.21.15.81:0/1237091224 wait complete. 2024-05-27T06:50:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 Processor -- start 2024-05-27T06:50:40.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- start start 2024-05-27T06:50:40.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1344072430 0x7f1344157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13441f9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f13440ce6f0 con 0x7f1344161cd0 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f13440c7dd0 con 0x7f1344072430 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f13440c5610 con 0x7f13440c5b60 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34600/0 (socket says 172.21.15.81:34600) 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 -- 172.21.15.81:0/776392843 learned_addr learned my addr 172.21.15.81:0/776392843 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1341e00640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13441f9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1342800640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1344072430 0x7f1344157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 -- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 msgr2=0x7f13441f9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13441f9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 -- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1344072430 msgr2=0x7f1344157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1344072430 0x7f1344157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 -- 172.21.15.81:0/776392843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13441f9e60 con 0x7f1344161cd0 2024-05-27T06:50:40.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1343200640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f133c009d90 tx=0x7f133c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:40.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0b2050 con 0x7f1344161cd0 2024-05-27T06:50:40.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f133c0b3040 con 0x7f1344161cd0 2024-05-27T06:50:40.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c003ba0 con 0x7f1344161cd0 2024-05-27T06:50:40.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134414b600 con 0x7f1344161cd0 2024-05-27T06:50:40.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.372+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134414b8a0 con 0x7f1344161cd0 2024-05-27T06:50:40.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134406a560 con 0x7f1344161cd0 2024-05-27T06:50:40.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f133c0ad650 con 0x7f1344161cd0 2024-05-27T06:50:40.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f133aa00640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 0x7f1324071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:40.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7f133c129bc0 con 0x7f1344161cd0 2024-05-27T06:50:40.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f1342800640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 0x7f1324071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:40.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.376+0000 7f1342800640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 0x7f1324071910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1334007a70 tx=0x7f13340079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:40.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.380+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f133c016560 con 0x7f1344161cd0 2024-05-27T06:50:40.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13440c5940 con 0x7f1344161cd0 2024-05-27T06:50:40.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f133aa00640 1 -- 172.21.15.81:0/776392843 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7f133c0f6090 con 0x7f1344161cd0 2024-05-27T06:50:40.567 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:40.567 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:40.567 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 msgr2=0x7f1324071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 0x7f1324071910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1334007a70 tx=0x7f13340079c0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 msgr2=0x7f13441f9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f133c009d90 tx=0x7f133c019710 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 shutdown_connections 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f13440c5b60 0x7f13441f9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f132406f400 0x7f1324071910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1344161cd0 0x7f13441f9860 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 --2- 172.21.15.81:0/776392843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1344072430 0x7f1344157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 >> 172.21.15.81:0/776392843 conn(0x7f1344070700 msgr2=0x7f1344074fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 shutdown_connections 2024-05-27T06:50:40.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:40.572+0000 7f1349400640 1 -- 172.21.15.81:0/776392843 wait complete. 2024-05-27T06:50:40.588 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:42.589 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:42.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/29' 2024-05-27T06:50:42.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/32' 2024-05-27T06:50:42.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/22' 2024-05-27T06:50:42.651 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/25' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/30' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/33' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/23' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/35' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/26' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './226/37' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './226' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/21' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/27' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/31' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/36' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/20' 2024-05-27T06:50:42.652 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/39' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/34' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/38' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/28' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/24' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/29' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/32' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/22' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/25' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/30' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/33' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/23' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/35' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/26' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './227/37' 2024-05-27T06:50:42.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './227' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/21' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/27' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/31' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/36' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/20' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/39' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/34' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/38' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/28' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/24' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/29' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/32' 2024-05-27T06:50:42.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/22' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/25' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/30' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/33' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/23' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/35' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/26' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './228/37' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './228' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/21' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/27' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/31' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/36' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/20' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/39' 2024-05-27T06:50:42.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/34' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/38' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/28' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/24' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/29' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/32' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/22' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/25' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/30' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/33' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/23' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/35' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/26' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './229/37' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './229' 2024-05-27T06:50:42.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/21' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/27' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/31' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/36' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/20' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/39' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/34' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/38' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/28' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/24' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/29' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/32' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/22' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/25' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/30' 2024-05-27T06:50:42.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/33' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/23' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/35' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/26' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './230/37' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './230' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/21' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/27' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/31' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/36' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/20' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/39' 2024-05-27T06:50:42.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/34' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/38' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/28' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/24' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/29' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/32' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/22' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/25' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/30' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/33' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/23' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/35' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/26' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './231/37' 2024-05-27T06:50:42.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './231' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/21' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/27' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/31' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/36' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/20' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/39' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/34' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/38' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/28' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/24' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/29' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/32' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/22' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/25' 2024-05-27T06:50:42.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/30' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/33' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/23' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/35' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/26' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './232/37' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './232' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/21' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/27' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/31' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/36' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/20' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/39' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/34' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/38' 2024-05-27T06:50:42.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/28' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/24' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/29' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/32' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/22' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/25' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/30' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/33' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/23' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/35' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/26' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './233/37' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './233' 2024-05-27T06:50:42.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/21' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/27' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/31' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/36' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/20' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/39' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/34' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/38' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/28' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/24' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/29' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/32' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/22' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/25' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/30' 2024-05-27T06:50:42.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/33' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/23' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/35' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/26' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './234/37' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './234' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/21' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/27' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/31' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/36' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/20' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/39' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/34' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/38' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/28' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/24' 2024-05-27T06:50:42.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/29' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/32' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/22' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/25' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/30' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/33' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/23' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/35' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/26' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './235/37' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './235' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/21' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/27' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/31' 2024-05-27T06:50:42.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/36' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/20' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/39' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/34' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/38' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/28' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/24' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/29' 2024-05-27T06:50:42.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/32' 2024-05-27T06:50:42.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 Processor -- start 2024-05-27T06:50:42.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 -- start start 2024-05-27T06:50:42.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 0x7fd7d41603e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd7d4160920 0x7fd7d40ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d406eb50 con 0x7fd7d415c7d0 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d415cc80 con 0x7fd7d4160920 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7dc000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d415cf60 con 0x7fd7d4156cd0 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d9600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd7d415c7d0 0x7fd7d40cfe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39612/0 (socket says 172.21.15.81:39612) 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d9600640 1 -- 172.21.15.81:0/2264706880 learned_addr learned my addr 172.21.15.81:0/2264706880 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7da000640 1 --2- 172.21.15.81:0/2264706880 >> v2:172.21.15.106:3300/0 conn(0x7fd7d4160920 0x7fd7d40ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d8c00640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 0x7fd7d41603e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d3e00640 1 -- 172.21.15.81:0/2264706880 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 319630082 0 0) 0x7fd7d406eb50 con 0x7fd7d415c7d0 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d3e00640 1 -- 172.21.15.81:0/2264706880 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7bc003530 con 0x7fd7d415c7d0 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7da000640 1 -- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 msgr2=0x7fd7d41603e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7da000640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 0x7fd7d41603e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7da000640 1 -- 172.21.15.81:0/2264706880 >> v1:172.21.15.81:6789/0 conn(0x7fd7d415c7d0 legacy=0x7fd7d40cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7da000640 1 -- 172.21.15.81:0/2264706880 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d415d240 con 0x7fd7d4160920 2024-05-27T06:50:42.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.760+0000 7fd7d8c00640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 0x7fd7d41603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:50:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.764+0000 7fd7da000640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d40ca2e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009d90 tx=0x7fd7cc01f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42fe6e3b8ff6e35 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.764+0000 7fd7d3e00640 1 -- 172.21.15.81:0/2264706880 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0b4070 con 0x7fd7d4160920 2024-05-27T06:50:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.764+0000 7fd7d3e00640 1 -- 172.21.15.81:0/2264706880 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc0af600 con 0x7fd7d4160920 2024-05-27T06:50:42.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.764+0000 7fd7d3e00640 1 -- 172.21.15.81:0/2264706880 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0af9d0 con 0x7fd7d4160920 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 msgr2=0x7fd7d40ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d40ca2e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009d90 tx=0x7fd7cc01f150 comp rx=0 tx=0).stop 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- 172.21.15.81:0/2264706880 shutdown_connections 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d40ca2e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2264706880 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d4156cd0 0x7fd7d41603e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- 172.21.15.81:0/2264706880 >> 172.21.15.81:0/2264706880 conn(0x7fd7d4068ae0 msgr2=0x7fd7d4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:42.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- 172.21.15.81:0/2264706880 shutdown_connections 2024-05-27T06:50:42.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- 172.21.15.81:0/2264706880 wait complete. 2024-05-27T06:50:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 Processor -- start 2024-05-27T06:50:42.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.768+0000 7fd7dc000640 1 -- start start 2024-05-27T06:50:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d415c7d0 0x7fd7d41544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d4074850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d40d0570 con 0x7fd7d4156cd0 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d415fec0 con 0x7fd7d4160920 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d40d06f0 con 0x7fd7d415c7d0 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34614/0 (socket says 172.21.15.81:34614) 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d9600640 1 -- 172.21.15.81:0/2590029607 learned_addr learned my addr 172.21.15.81:0/2590029607 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d4074850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d8c00640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d415c7d0 0x7fd7d41544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 -- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d415c7d0 msgr2=0x7fd7d41544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d415c7d0 0x7fd7d41544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 -- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 msgr2=0x7fd7d415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 -- 172.21.15.81:0/2590029607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d4154a00 con 0x7fd7d4160920 2024-05-27T06:50:42.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d9600640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7da000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d4074850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0af000 tx=0x7fd7cc01bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0c0020 con 0x7fd7d4160920 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc0ba470 con 0x7fd7d4160920 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d4075250 con 0x7fd7d4160920 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0b4040 con 0x7fd7d4160920 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d40c8b00 con 0x7fd7d4160920 2024-05-27T06:50:42.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.772+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d406a560 con 0x7fd7d4160920 2024-05-27T06:50:42.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.776+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7cc0c3070 con 0x7fd7d4160920 2024-05-27T06:50:42.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.776+0000 7fd7d1600640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 0x7fd7b4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:42.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.776+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7fd7cc12cf30 con 0x7fd7d4160920 2024-05-27T06:50:42.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.776+0000 7fd7d9600640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 0x7fd7b4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:42.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.776+0000 7fd7d9600640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 0x7fd7b4071910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8007fd0 tx=0x7fd7c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:42.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.780+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd7cc018cb0 con 0x7fd7d4160920 2024-05-27T06:50:42.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7d40c85b0 con 0x7fd7d4160920 2024-05-27T06:50:42.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7d1600640 1 -- 172.21.15.81:0/2590029607 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7fd7cc0f94a0 con 0x7fd7d4160920 2024-05-27T06:50:42.975 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:42.975 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:42.975 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 msgr2=0x7fd7b4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 0x7fd7b4071910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8007fd0 tx=0x7fd7c8009040 comp rx=0 tx=0).stop 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 msgr2=0x7fd7d4074850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d4074850 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0af000 tx=0x7fd7cc01bef0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 shutdown_connections 2024-05-27T06:50:42.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7d415c7d0 0x7fd7d41544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7b406f400 0x7fd7b4071910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7d4160920 0x7fd7d4074850 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 --2- 172.21.15.81:0/2590029607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7d4156cd0 0x7fd7d415a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 >> 172.21.15.81:0/2590029607 conn(0x7fd7d4068ae0 msgr2=0x7fd7d40ca270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 shutdown_connections 2024-05-27T06:50:42.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:42.980+0000 7fd7dc000640 1 -- 172.21.15.81:0/2590029607 wait complete. 2024-05-27T06:50:42.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:44.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:45.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.164+0000 7f4a1e000640 1 Processor -- start 2024-05-27T06:50:45.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.164+0000 7f4a1e000640 1 -- start start 2024-05-27T06:50:45.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1e000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4a18165f80 0x7f4a181683a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:45.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1e000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a180ca180 con 0x7f4a18156920 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1e000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a180ccfe0 con 0x7f4a180ccbd0 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1e000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4a180ca700 con 0x7f4a18165f80 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1cc00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4a18156920 0x7f4a180cfcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39616/0 (socket says 172.21.15.81:39616) 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1cc00640 1 -- 172.21.15.81:0/4095365860 learned_addr learned my addr 172.21.15.81:0/4095365860 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1d600640 1 --2- 172.21.15.81:0/4095365860 >> v2:172.21.15.106:3300/0 conn(0x7f4a18165f80 0x7f4a181683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988820769 0 0) 0x7f4a180ca180 con 0x7f4a18156920 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a00003530 con 0x7f4a18156920 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233961799 0 0) 0x7f4a180ccfe0 con 0x7f4a180ccbd0 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a180ca180 con 0x7f4a180ccbd0 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1d600640 1 -- 172.21.15.81:0/4095365860 >> v1:172.21.15.106:6789/0 conn(0x7f4a180ccbd0 legacy=0x7f4a18160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1d600640 1 -- 172.21.15.81:0/4095365860 >> v1:172.21.15.81:6789/0 conn(0x7f4a18156920 legacy=0x7f4a180cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645814115 0 0) 0x7f4a00003530 con 0x7f4a18156920 2024-05-27T06:50:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.168+0000 7f4a1d600640 1 -- 172.21.15.81:0/4095365860 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a180ca420 con 0x7f4a18165f80 2024-05-27T06:50:45.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1d600640 1 --2- 172.21.15.81:0/4095365860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a18165f80 0x7f4a181683a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a180cd1b0 tx=0x7f4a1001c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b91ebc8fbd62fcb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:45.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a100b2070 con 0x7f4a18165f80 2024-05-27T06:50:45.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a100a5530 con 0x7f4a18165f80 2024-05-27T06:50:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a17400640 1 -- 172.21.15.81:0/4095365860 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c001720 con 0x7f4a18165f80 2024-05-27T06:50:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 -- 172.21.15.81:0/4095365860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a18165f80 msgr2=0x7f4a181683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 --2- 172.21.15.81:0/4095365860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a18165f80 0x7f4a181683a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a180cd1b0 tx=0x7f4a1001c7e0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 -- 172.21.15.81:0/4095365860 shutdown_connections 2024-05-27T06:50:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 --2- 172.21.15.81:0/4095365860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a18165f80 0x7f4a181683a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 -- 172.21.15.81:0/4095365860 >> 172.21.15.81:0/4095365860 conn(0x7f4a18068ab0 msgr2=0x7f4a18068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:45.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 -- 172.21.15.81:0/4095365860 shutdown_connections 2024-05-27T06:50:45.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 -- 172.21.15.81:0/4095365860 wait complete. 2024-05-27T06:50:45.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.172+0000 7f4a1e000640 1 Processor -- start 2024-05-27T06:50:45.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- start start 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a180ccbd0 0x7f4a18167960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 0x7f4a1815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4a18156d30 con 0x7f4a18165f80 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1806e980 con 0x7f4a180ccbd0 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4a18072b80 con 0x7f4a18156920 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34630/0 (socket says 172.21.15.81:34630) 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 -- 172.21.15.81:0/1852423885 learned_addr learned my addr 172.21.15.81:0/1852423885 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1cc00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a180ccbd0 0x7f4a18167960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a17e00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 0x7f4a1815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 -- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 msgr2=0x7f4a1815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 0x7f4a1815ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 -- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a180ccbd0 msgr2=0x7f4a18167960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a180ccbd0 0x7f4a18167960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 -- 172.21.15.81:0/1852423885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a18167ea0 con 0x7f4a18165f80 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a17e00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 0x7f4a1815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:50:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1d600640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a100193f0 tx=0x7f4a10019580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a100bd020 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a100b8470 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a100b2040 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a180c7600 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a180c8b70 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.176+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a18050870 con 0x7f4a18165f80 2024-05-27T06:50:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.180+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4a100c0020 con 0x7f4a18165f80 2024-05-27T06:50:45.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.180+0000 7f4a14c00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 0x7f49f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:45.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.180+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7f4a1012a280 con 0x7f4a18165f80 2024-05-27T06:50:45.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.180+0000 7f4a1cc00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 0x7f49f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:45.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.180+0000 7f4a1cc00640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 0x7f49f8071910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001890 tx=0x7f4a0c001620 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:45.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.184+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4a100c7030 con 0x7f4a18165f80 2024-05-27T06:50:45.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a1806de60 con 0x7f4a18165f80 2024-05-27T06:50:45.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a14c00640 1 -- 172.21.15.81:0/1852423885 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7f4a10016460 con 0x7f4a18165f80 2024-05-27T06:50:45.371 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:45.371 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:45.371 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:45.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 msgr2=0x7f49f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 0x7f49f8071910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c001890 tx=0x7f4a0c001620 comp rx=0 tx=0).stop 2024-05-27T06:50:45.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 msgr2=0x7f4a181f9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a100193f0 tx=0x7f4a10019580 comp rx=0 tx=0).stop 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 shutdown_connections 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a18156920 0x7f4a1815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49f806f400 0x7f49f8071910 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a18165f80 0x7f4a181f9710 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 --2- 172.21.15.81:0/1852423885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a180ccbd0 0x7f4a18167960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 >> 172.21.15.81:0/1852423885 conn(0x7f4a18068ab0 msgr2=0x7f4a1815bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 shutdown_connections 2024-05-27T06:50:45.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:45.376+0000 7f4a1e000640 1 -- 172.21.15.81:0/1852423885 wait complete. 2024-05-27T06:50:45.392 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/22' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/25' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/30' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/33' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/23' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/35' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/26' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './236/37' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './236' 2024-05-27T06:50:45.589 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/21' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/27' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/31' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/36' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/20' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/39' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/34' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/38' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/28' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/24' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/29' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/32' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/22' 2024-05-27T06:50:45.590 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/25' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/30' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/33' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/23' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/35' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/26' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './237/37' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './237' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/21' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/27' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/31' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/36' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/20' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/39' 2024-05-27T06:50:45.591 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/34' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/38' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/28' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/24' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/29' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/32' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/22' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/25' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/30' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/33' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/23' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/35' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/26' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed './238/37' 2024-05-27T06:50:45.592 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './238' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/21' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/27' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/31' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/36' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/20' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/39' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/34' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/38' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/28' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/24' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/29' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/32' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/22' 2024-05-27T06:50:45.593 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/25' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/30' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/33' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/23' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/35' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/26' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './239/37' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './239' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/21' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/27' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/31' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/36' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/20' 2024-05-27T06:50:45.594 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/39' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/34' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/38' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/28' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/24' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/29' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/32' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/22' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/25' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/30' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/33' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/23' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/35' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/26' 2024-05-27T06:50:45.595 INFO:tasks.workunit.client.0.smithi081.stdout:removed './240/37' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './240' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/21' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/27' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/31' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/36' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/20' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/39' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/34' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/38' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/28' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/24' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/29' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/32' 2024-05-27T06:50:45.596 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/22' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/25' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/30' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/33' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/23' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/35' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/26' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './241/37' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './241' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/21' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/27' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/31' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/36' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/20' 2024-05-27T06:50:45.597 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/39' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/34' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/38' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/28' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/24' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/29' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/32' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/22' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/25' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/30' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/33' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/23' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/35' 2024-05-27T06:50:45.598 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/26' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './242/37' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './242' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/21' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/27' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/31' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/36' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/20' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/39' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/34' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/38' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/28' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/24' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/29' 2024-05-27T06:50:45.599 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/32' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/22' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/25' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/30' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/33' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/23' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/35' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/26' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './243/37' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './243' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/21' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/27' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/31' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/36' 2024-05-27T06:50:45.600 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/20' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/39' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/34' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/38' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/28' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/24' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/29' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/32' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/22' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/25' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/30' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/33' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/23' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/35' 2024-05-27T06:50:45.601 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/26' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './244/37' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './244' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/21' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/27' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/31' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/36' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/20' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/39' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/34' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/38' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/28' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/24' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/29' 2024-05-27T06:50:45.602 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/32' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/22' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/25' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/30' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/33' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/23' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/35' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/26' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './245/37' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './245' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/21' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/27' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/31' 2024-05-27T06:50:45.603 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/36' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/20' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/39' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/34' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/38' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/28' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/24' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/29' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/32' 2024-05-27T06:50:45.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/22' 2024-05-27T06:50:46.225 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/25' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/30' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/33' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/23' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/35' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/26' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './246/37' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './246' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/21' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/27' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/31' 2024-05-27T06:50:46.226 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/36' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/20' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/39' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/34' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/38' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/28' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/24' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/29' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/32' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/22' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/25' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/30' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/33' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/23' 2024-05-27T06:50:46.227 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/35' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/26' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './247/37' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './247' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/21' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/27' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/31' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/36' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/20' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/39' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/34' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/38' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/28' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/24' 2024-05-27T06:50:46.228 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/29' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/32' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/22' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/25' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/30' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/33' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/23' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/35' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/26' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './248/37' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './248' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/21' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/27' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/31' 2024-05-27T06:50:46.229 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/36' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/20' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/39' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/34' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/38' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/28' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/24' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/29' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/32' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/22' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/25' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/30' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/33' 2024-05-27T06:50:46.230 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/23' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/35' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/26' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './249/37' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './249' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/21' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/27' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/31' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/36' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/20' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/39' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/34' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/38' 2024-05-27T06:50:46.231 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/28' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/24' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/29' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/32' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/22' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/25' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/30' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/33' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/23' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/35' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/26' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './250/37' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './250' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/21' 2024-05-27T06:50:46.232 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/27' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/31' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/36' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/20' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/39' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/34' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/38' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/28' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/24' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/29' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/32' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/22' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/25' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/30' 2024-05-27T06:50:46.233 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/33' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/23' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/35' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/26' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './251/37' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './251' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/21' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/27' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/31' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/36' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/20' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/39' 2024-05-27T06:50:46.234 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/34' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/38' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/28' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/24' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/29' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/32' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/22' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/25' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/30' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/33' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/23' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/35' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/26' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './252/37' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './252' 2024-05-27T06:50:46.235 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/21' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/27' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/31' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/36' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/20' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/39' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/34' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/38' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/28' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/24' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/29' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/32' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/22' 2024-05-27T06:50:46.236 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/25' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/30' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/33' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/23' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/35' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/26' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './253/37' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './253' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/21' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/27' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/31' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/36' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/20' 2024-05-27T06:50:46.237 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/39' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/34' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/38' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/28' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/24' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/29' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/32' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/22' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/25' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/30' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/33' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/23' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/35' 2024-05-27T06:50:46.238 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/26' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './254/37' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './254' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/21' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/27' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/31' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/36' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/20' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/39' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/34' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/38' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/28' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/24' 2024-05-27T06:50:46.239 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/29' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/32' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/22' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/25' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/30' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/33' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/23' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/35' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/26' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './255/37' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './255' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/21' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/27' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/31' 2024-05-27T06:50:46.240 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/36' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/20' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/39' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/34' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/38' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/28' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/24' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/29' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/32' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/22' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/25' 2024-05-27T06:50:46.241 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/30' 2024-05-27T06:50:47.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:47.549 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/33' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/23' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/35' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/26' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './256/37' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './256' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/21' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/27' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/31' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/36' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/20' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/39' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/34' 2024-05-27T06:50:47.550 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/38' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/28' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/24' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/29' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/32' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/22' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/25' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/30' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/33' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/23' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/35' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/26' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './257/37' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './257' 2024-05-27T06:50:47.551 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/21' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/27' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/31' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/36' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/20' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/39' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/34' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/38' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/28' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/24' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/29' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/32' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/22' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/25' 2024-05-27T06:50:47.552 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/30' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/33' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/23' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/35' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/26' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './258/37' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './258' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/21' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/27' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/31' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/36' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/20' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/39' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/34' 2024-05-27T06:50:47.553 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/38' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/28' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/24' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/29' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/32' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/22' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/25' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/30' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/33' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/23' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/35' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/26' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed './259/37' 2024-05-27T06:50:47.554 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './259' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/21' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/27' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/31' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/36' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/20' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/39' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/34' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/38' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/28' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/24' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/29' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/32' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/22' 2024-05-27T06:50:47.555 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/25' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/30' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/33' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/23' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/35' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/26' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './260/37' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './260' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/21' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/27' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/31' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/36' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/20' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/39' 2024-05-27T06:50:47.556 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/34' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/38' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/28' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/24' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/29' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/32' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/22' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/25' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/30' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/33' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/23' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/35' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/26' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed './261/37' 2024-05-27T06:50:47.557 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './261' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/21' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/27' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/31' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/36' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/20' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/39' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/34' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/38' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/28' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/24' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/29' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/32' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/22' 2024-05-27T06:50:47.558 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/25' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/30' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/33' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/23' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/35' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/26' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './262/37' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './262' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/21' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/27' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/31' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/36' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/20' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/39' 2024-05-27T06:50:47.559 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/34' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/38' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/28' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/24' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/29' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/32' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/22' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/25' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/30' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/33' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/23' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/35' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/26' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed './263/37' 2024-05-27T06:50:47.560 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './263' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/21' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/27' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/31' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/36' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/20' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/39' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/34' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/38' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/28' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/24' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/29' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/32' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/22' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/25' 2024-05-27T06:50:47.561 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/30' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/33' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/23' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/35' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/26' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './264/37' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './264' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/21' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/27' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/31' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/36' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/20' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/39' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/34' 2024-05-27T06:50:47.562 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/38' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/28' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/24' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/29' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/32' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/22' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/25' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/30' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/33' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/23' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/35' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/26' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './265/37' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './265' 2024-05-27T06:50:47.563 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/21' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/27' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/31' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/36' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/20' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/39' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/34' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/38' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/28' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/24' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/29' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/32' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/22' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/25' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/30' 2024-05-27T06:50:47.564 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/33' 2024-05-27T06:50:47.565 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/23' 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.564+0000 7fa2e5a00640 1 Processor -- start 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.564+0000 7fa2e5a00640 1 -- start start 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2e5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa2e015a920 0x7fa2e0165ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2e5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e00ca630 con 0x7fa2e00cbce0 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2e5a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e00c7230 con 0x7fa2e015a510 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2e5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e00cadf0 con 0x7fa2e015a920 2024-05-27T06:50:47.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2dea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa2e00cbce0 0x7fa2e00c99c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33502/0 (socket says 172.21.15.81:33502) 2024-05-27T06:50:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2dea00640 1 -- 172.21.15.81:0/3481948435 learned_addr learned my addr 172.21.15.81:0/3481948435 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2df400640 1 --2- 172.21.15.81:0/3481948435 >> v2:172.21.15.81:3300/0 conn(0x7fa2e015a920 0x7fa2e0165ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2df400640 1 -- 172.21.15.81:0/3481948435 >> v1:172.21.15.106:6789/0 conn(0x7fa2e015a510 legacy=0x7fa2e00746b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2df400640 1 -- 172.21.15.81:0/3481948435 >> v1:172.21.15.81:6789/0 conn(0x7fa2e00cbce0 legacy=0x7fa2e00c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.568+0000 7fa2df400640 1 -- 172.21.15.81:0/3481948435 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e00c7510 con 0x7fa2e015a920 2024-05-27T06:50:47.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2df400640 1 --2- 172.21.15.81:0/3481948435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e015a920 0x7fa2e0165ea0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009d90 tx=0x7fa2d801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f997d4e757d9d393 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2dd600640 1 -- 172.21.15.81:0/3481948435 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d80b4070 con 0x7fa2e015a920 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2dd600640 1 -- 172.21.15.81:0/3481948435 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d80af600 con 0x7fa2e015a920 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2dd600640 1 -- 172.21.15.81:0/3481948435 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d80af9d0 con 0x7fa2e015a920 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3481948435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e015a920 msgr2=0x7fa2e0165ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3481948435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e015a920 0x7fa2e0165ea0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009d90 tx=0x7fa2d801f150 comp rx=0 tx=0).stop 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3481948435 shutdown_connections 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3481948435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e015a920 0x7fa2e0165ea0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3481948435 >> 172.21.15.81:0/3481948435 conn(0x7fa2e0072aa0 msgr2=0x7fa2e00cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:47.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3481948435 shutdown_connections 2024-05-27T06:50:47.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.572+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3481948435 wait complete. 2024-05-27T06:50:47.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 Processor -- start 2024-05-27T06:50:47.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- start start 2024-05-27T06:50:47.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:47.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2e015a510 0x7fa2e014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:47.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e015a920 0x7fa2e014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e015ad30 con 0x7fa2e00cbce0 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e00ca0d0 con 0x7fa2e015a920 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e00c77f0 con 0x7fa2e015a510 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55432/0 (socket says 172.21.15.81:55432) 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 -- 172.21.15.81:0/3828980890 learned_addr learned my addr 172.21.15.81:0/3828980890 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2de000640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2e015a510 0x7fa2e014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2df400640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e015a920 0x7fa2e014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 -- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2e015a510 msgr2=0x7fa2e014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2e015a510 0x7fa2e014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 -- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e015a920 msgr2=0x7fa2e014e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e015a920 0x7fa2e014e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 -- 172.21.15.81:0/3828980890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e0154300 con 0x7fa2e00cbce0 2024-05-27T06:50:47.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2dea00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c460 tx=0x7fa2d000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:47.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0017070 con 0x7fa2e00cbce0 2024-05-27T06:50:47.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d0005d10 con 0x7fa2e00cbce0 2024-05-27T06:50:47.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e01f2d80 con 0x7fa2e00cbce0 2024-05-27T06:50:47.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00072b0 con 0x7fa2e00cbce0 2024-05-27T06:50:47.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.576+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e01f3040 con 0x7fa2e00cbce0 2024-05-27T06:50:47.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.580+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e006a560 con 0x7fa2e00cbce0 2024-05-27T06:50:47.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.580+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa2d001a070 con 0x7fa2e00cbce0 2024-05-27T06:50:47.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.580+0000 7fa2d6a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 0x7fa2c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:47.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.580+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6665+0+0 (secure 0 0 0) 0x7fa2d008e8f0 con 0x7fa2e00cbce0 2024-05-27T06:50:47.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.580+0000 7fa2de000640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 0x7fa2c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:47.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.584+0000 7fa2de000640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 0x7fa2c0071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa2e01f73d0 tx=0x7fa2cc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:47.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.584+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa2d005adc0 con 0x7fa2e00cbce0 2024-05-27T06:50:47.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.804+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2e00c94c0 con 0x7fa2e00cbce0 2024-05-27T06:50:47.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2d6a00640 1 -- 172.21.15.81:0/3828980890 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6903 (secure 0 0 0) 0x7fa2d005f0c0 con 0x7fa2e00cbce0 2024-05-27T06:50:47.802 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:47.802 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5049,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:48:34.891495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":4281,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"c","rank":3,"incarnation":14,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6835/2990302599","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2990302599},{"type":"v1","addr":"172.21.15.81:6835","nonce":2990302599}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285,4281]},"id":1}]} 2024-05-27T06:50:47.802 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 29 2024-05-27T06:50:47.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 msgr2=0x7fa2c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 0x7fa2c0071910 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa2e01f73d0 tx=0x7fa2cc009040 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 msgr2=0x7fa2e0145970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c460 tx=0x7fa2d000c920 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 shutdown_connections 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa2e015a510 0x7fa2e014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa2c006f400 0x7fa2c0071910 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e015a920 0x7fa2e014e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 --2- 172.21.15.81:0/3828980890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e00cbce0 0x7fa2e0145970 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 >> 172.21.15.81:0/3828980890 conn(0x7fa2e0072aa0 msgr2=0x7fa2e00745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 shutdown_connections 2024-05-27T06:50:47.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:47.808+0000 7fa2e5a00640 1 -- 172.21.15.81:0/3828980890 wait complete. 2024-05-27T06:50:47.826 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-05-27T06:50:49.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/35' 2024-05-27T06:50:49.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/26' 2024-05-27T06:50:49.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './266/37' 2024-05-27T06:50:49.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './266' 2024-05-27T06:50:49.604 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/21' 2024-05-27T06:50:49.605 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/27' 2024-05-27T06:50:49.605 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/31' 2024-05-27T06:50:49.605 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/36' 2024-05-27T06:50:49.605 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/20' 2024-05-27T06:50:49.605 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/39' 2024-05-27T06:50:49.606 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/34' 2024-05-27T06:50:49.606 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/38' 2024-05-27T06:50:49.606 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/28' 2024-05-27T06:50:49.606 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/24' 2024-05-27T06:50:49.606 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/29' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/32' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/22' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/25' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/30' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/33' 2024-05-27T06:50:49.607 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/23' 2024-05-27T06:50:49.608 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/35' 2024-05-27T06:50:49.608 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/26' 2024-05-27T06:50:49.608 INFO:tasks.workunit.client.0.smithi081.stdout:removed './267/37' 2024-05-27T06:50:49.608 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './267' 2024-05-27T06:50:49.608 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/21' 2024-05-27T06:50:49.609 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/27' 2024-05-27T06:50:49.609 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/31' 2024-05-27T06:50:49.609 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/36' 2024-05-27T06:50:49.609 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/20' 2024-05-27T06:50:49.609 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/39' 2024-05-27T06:50:49.610 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/34' 2024-05-27T06:50:49.610 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/38' 2024-05-27T06:50:49.610 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/28' 2024-05-27T06:50:49.610 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/24' 2024-05-27T06:50:49.610 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/29' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/32' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/22' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/25' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/30' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/33' 2024-05-27T06:50:49.611 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/23' 2024-05-27T06:50:49.612 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/35' 2024-05-27T06:50:49.612 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/26' 2024-05-27T06:50:49.612 INFO:tasks.workunit.client.0.smithi081.stdout:removed './268/37' 2024-05-27T06:50:49.612 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './268' 2024-05-27T06:50:49.612 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/21' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/27' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/31' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/36' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/20' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/39' 2024-05-27T06:50:49.613 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/34' 2024-05-27T06:50:49.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/38' 2024-05-27T06:50:49.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/28' 2024-05-27T06:50:49.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/24' 2024-05-27T06:50:49.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/29' 2024-05-27T06:50:49.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/32' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/22' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/25' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/30' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/33' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/23' 2024-05-27T06:50:49.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/35' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/26' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './269/37' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './269' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/21' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/27' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/31' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/36' 2024-05-27T06:50:49.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/20' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/39' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/34' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/38' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/28' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/24' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/29' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/32' 2024-05-27T06:50:49.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/22' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/25' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/30' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/33' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/23' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/35' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/26' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './270/37' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './270' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/21' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/27' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/31' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/36' 2024-05-27T06:50:49.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/20' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/39' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/34' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/38' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/28' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/24' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/29' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/32' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/22' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/25' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/30' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/33' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/23' 2024-05-27T06:50:49.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/35' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/26' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './271/37' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './271' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/21' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/27' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/31' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/36' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/20' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/39' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/34' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/38' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/28' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/24' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/29' 2024-05-27T06:50:49.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/32' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/22' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/25' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/30' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/33' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/23' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/35' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/26' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './272/37' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './272' 2024-05-27T06:50:49.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/21' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/27' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/31' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/36' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/20' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/39' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/34' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/38' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/28' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/24' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/29' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/32' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/22' 2024-05-27T06:50:49.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/25' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/30' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/33' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/23' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/35' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/26' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './273/37' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './273' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/21' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/27' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/31' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/36' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/20' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/39' 2024-05-27T06:50:49.623 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/34' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/38' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/28' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/24' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/29' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/32' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/22' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/25' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/30' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/33' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/23' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/35' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/26' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed './274/37' 2024-05-27T06:50:49.624 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './274' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/21' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/27' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/31' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/36' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/20' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/39' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/34' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/38' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/28' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/24' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/29' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/32' 2024-05-27T06:50:49.625 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/22' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/25' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/30' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/33' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/23' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/35' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/26' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './275/37' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './275' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/21' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/27' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/31' 2024-05-27T06:50:49.626 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/36' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/20' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/39' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/34' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/38' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/28' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/24' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/29' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/32' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/22' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/25' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/30' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/33' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/23' 2024-05-27T06:50:49.627 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/35' 2024-05-27T06:50:49.628 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/26' 2024-05-27T06:50:49.826 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:49.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:49.996+0000 7f8141600640 1 Processor -- start 2024-05-27T06:50:49.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:49.996+0000 7f8141600640 1 -- start start 2024-05-27T06:50:49.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8141600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f813c0c9630 0x7f813c154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:49.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8141600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f813c15ded0 0x7f813c164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8141600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f813c0c9c70 con 0x7f813c15dac0 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8141600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f813c0c9a40 con 0x7f813c0c9630 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8141600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f813c15b0c0 con 0x7f813c15ded0 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f813c0c9630 0x7f813c154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813b400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f813c15dac0 0x7f813c072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33508/0 (socket says 172.21.15.81:33508) 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f813c0c9630 0x7f813c154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55436/0 (socket says 172.21.15.81:55436) 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813b400640 1 -- 172.21.15.81:0/2683045834 learned_addr learned my addr 172.21.15.81:0/2683045834 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f8140c00640 1 --2- 172.21.15.81:0/2683045834 >> v2:172.21.15.106:3300/0 conn(0x7f813c15ded0 0x7f813c164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 -- 172.21.15.81:0/2683045834 >> v2:172.21.15.106:3300/0 conn(0x7f813c15ded0 msgr2=0x7f813c164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 --2- 172.21.15.81:0/2683045834 >> v2:172.21.15.106:3300/0 conn(0x7f813c15ded0 0x7f813c164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 -- 172.21.15.81:0/2683045834 >> v1:172.21.15.81:6789/0 conn(0x7f813c15dac0 legacy=0x7f813c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:49.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.000+0000 7f813be00640 1 -- 172.21.15.81:0/2683045834 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813c15a4d0 con 0x7f813c0c9630 2024-05-27T06:50:49.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f813be00640 1 --2- 172.21.15.81:0/2683045834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c154780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f813c0c3730 tx=0x7f812c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7191057a09f3efa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:49.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f813aa00640 1 -- 172.21.15.81:0/2683045834 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c0ab5f0 con 0x7f813c0c9630 2024-05-27T06:50:49.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f813aa00640 1 -- 172.21.15.81:0/2683045834 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812c0b0070 con 0x7f813c0c9630 2024-05-27T06:50:49.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f813aa00640 1 -- 172.21.15.81:0/2683045834 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c0ab9b0 con 0x7f813c0c9630 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- 172.21.15.81:0/2683045834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 msgr2=0x7f813c154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 --2- 172.21.15.81:0/2683045834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c154780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f813c0c3730 tx=0x7f812c01c3f0 comp rx=0 tx=0).stop 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- 172.21.15.81:0/2683045834 shutdown_connections 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 --2- 172.21.15.81:0/2683045834 >> v2:172.21.15.106:3300/0 conn(0x7f813c15ded0 0x7f813c164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 --2- 172.21.15.81:0/2683045834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c154780 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- 172.21.15.81:0/2683045834 >> 172.21.15.81:0/2683045834 conn(0x7f813c068880 msgr2=0x7f813c068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- 172.21.15.81:0/2683045834 shutdown_connections 2024-05-27T06:50:49.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- 172.21.15.81:0/2683045834 wait complete. 2024-05-27T06:50:50.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 Processor -- start 2024-05-27T06:50:50.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.004+0000 7f8141600640 1 -- start start 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f813c15dac0 0x7f813c1f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f813c15ded0 0x7f813c162b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f813c15e2e0 con 0x7f813c0c9630 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f813c0cd280 con 0x7f813c15dac0 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f813c072c30 con 0x7f813c15ded0 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55448/0 (socket says 172.21.15.81:55448) 2024-05-27T06:50:50.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 -- 172.21.15.81:0/2759810974 learned_addr learned my addr 172.21.15.81:0/2759810974 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813b400640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f813c15dac0 0x7f813c1f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8140c00640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f813c15ded0 0x7f813c162b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 -- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f813c15ded0 msgr2=0x7f813c162b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f813c15ded0 0x7f813c162b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 -- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f813c15dac0 msgr2=0x7f813c1f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f813c15dac0 0x7f813c1f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 -- 172.21.15.81:0/2759810974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813c1f9170 con 0x7f813c0c9630 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f813be00640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f812c0ad040 tx=0x7f812c01ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c0bf410 con 0x7f813c0c9630 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812c0bf5d0 con 0x7f813c0c9630 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813c1f9450 con 0x7f813c0c9630 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c0bf9c0 con 0x7f813c0c9630 2024-05-27T06:50:50.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f813c1f9a10 con 0x7f813c0c9630 2024-05-27T06:50:50.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.008+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f813c050870 con 0x7f813c0c9630 2024-05-27T06:50:50.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.012+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f812c0bd020 con 0x7f813c0c9630 2024-05-27T06:50:50.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.012+0000 7f8133e00640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 0x7f811c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.012+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f812c1295a0 con 0x7f813c0c9630 2024-05-27T06:50:50.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.012+0000 7f813b400640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 0x7f811c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.012+0000 7f813b400640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 0x7f811c071940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f813c1f6c90 tx=0x7f8128007ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:50.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.016+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f812c016070 con 0x7f813c0c9630 2024-05-27T06:50:50.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813c0cabe0 con 0x7f813c0c9630 2024-05-27T06:50:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8133e00640 1 -- 172.21.15.81:0/2759810974 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6074 (secure 0 0 0) 0x7f812c0f5d40 con 0x7f813c0c9630 2024-05-27T06:50:50.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:50.263 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:48.327819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:50:50.263 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 30 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 msgr2=0x7f811c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 0x7f811c071940 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f813c1f6c90 tx=0x7f8128007ef0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 msgr2=0x7f813c1507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f812c0ad040 tx=0x7f812c01ac30 comp rx=0 tx=0).stop 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 shutdown_connections 2024-05-27T06:50:50.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f813c15ded0 0x7f813c162b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f811c06f430 0x7f811c071940 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f813c15dac0 0x7f813c1f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 --2- 172.21.15.81:0/2759810974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f813c0c9630 0x7f813c1507c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 >> 172.21.15.81:0/2759810974 conn(0x7f813c068880 msgr2=0x7f813c072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 shutdown_connections 2024-05-27T06:50:50.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.268+0000 7f8141600640 1 -- 172.21.15.81:0/2759810974 wait complete. 2024-05-27T06:50:50.284 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-05-27T06:50:50.284 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:50:50.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:50.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.436+0000 7f9398a00640 1 Processor -- start 2024-05-27T06:50:50.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.436+0000 7f9398a00640 1 -- start start 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9398a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f939415fd20 0x7f9394156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9398a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f93940cb010 0x7f9394072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9398a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9394160130 con 0x7f93940c89a0 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9398a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f93940c9070 con 0x7f93940cb010 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9398a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9394074890 con 0x7f939415fd20 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f93940cb010 0x7f9394072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f93940cb010 0x7f9394072c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55450/0 (socket says 172.21.15.81:55450) 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 -- 172.21.15.81:0/3864009367 learned_addr learned my addr 172.21.15.81:0/3864009367 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:50.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9391e00640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f939415fd20 0x7f9394156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 -- 172.21.15.81:0/3864009367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f939415fd20 msgr2=0x7f9394156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:50.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f939415fd20 0x7f9394156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 -- 172.21.15.81:0/3864009367 >> v1:172.21.15.106:6789/0 conn(0x7f93940c89a0 legacy=0x7f93940ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.440+0000 7f9392800640 1 -- 172.21.15.81:0/3864009367 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93940c94c0 con 0x7f93940cb010 2024-05-27T06:50:50.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9392800640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93940cb010 0x7f9394072c10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e50 tx=0x7f938c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12a19a1214b7e0b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:50.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9391400640 1 -- 172.21.15.81:0/3864009367 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0a55f0 con 0x7f93940cb010 2024-05-27T06:50:50.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9391400640 1 -- 172.21.15.81:0/3864009367 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938c0a5790 con 0x7f93940cb010 2024-05-27T06:50:50.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9391400640 1 -- 172.21.15.81:0/3864009367 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938c0a5b60 con 0x7f93940cb010 2024-05-27T06:50:50.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- 172.21.15.81:0/3864009367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93940cb010 msgr2=0x7f9394072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93940cb010 0x7f9394072c10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f938c009e50 tx=0x7f938c01ca30 comp rx=0 tx=0).stop 2024-05-27T06:50:50.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- 172.21.15.81:0/3864009367 shutdown_connections 2024-05-27T06:50:50.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93940cb010 0x7f9394072c10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 --2- 172.21.15.81:0/3864009367 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f939415fd20 0x7f9394156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- 172.21.15.81:0/3864009367 >> 172.21.15.81:0/3864009367 conn(0x7f939406f230 msgr2=0x7f9394070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:50.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- 172.21.15.81:0/3864009367 shutdown_connections 2024-05-27T06:50:50.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- 172.21.15.81:0/3864009367 wait complete. 2024-05-27T06:50:50.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 Processor -- start 2024-05-27T06:50:50.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.444+0000 7f9398a00640 1 -- start start 2024-05-27T06:50:50.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93940c89a0 0x7f93941f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93940bf2b0 0x7f93940c0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93940c9760 con 0x7f939415fd20 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9394073fe0 con 0x7f93940bf2b0 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f93941698d0 con 0x7f93940c89a0 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55460/0 (socket says 172.21.15.81:55460) 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 -- 172.21.15.81:0/1346421228 learned_addr learned my addr 172.21.15.81:0/1346421228 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9391e00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93940c89a0 0x7f93941f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9392800640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93940bf2b0 0x7f93940c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 -- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93940c89a0 msgr2=0x7f93941f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93940c89a0 0x7f93941f6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 -- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93940bf2b0 msgr2=0x7f93940c0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93940bf2b0 0x7f93940c0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 -- 172.21.15.81:0/1346421228 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939415d6e0 con 0x7f939415fd20 2024-05-27T06:50:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9389400640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c3c0 tx=0x7f937c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c017070 con 0x7f939415fd20 2024-05-27T06:50:50.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f937c0052d0 con 0x7f939415fd20 2024-05-27T06:50:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93941f6550 con 0x7f939415fd20 2024-05-27T06:50:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0054c0 con 0x7f939415fd20 2024-05-27T06:50:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939415ce40 con 0x7f939415fd20 2024-05-27T06:50:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.448+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939406a560 con 0x7f939415fd20 2024-05-27T06:50:50.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.452+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f937c01a070 con 0x7f939415fd20 2024-05-27T06:50:50.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.452+0000 7f938aa00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 0x7f9370071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:50.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.452+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f937c08e6e0 con 0x7f939415fd20 2024-05-27T06:50:50.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.452+0000 7f9391e00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 0x7f9370071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:50.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.452+0000 7f9391e00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 0x7f9370071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9384009ac0 tx=0x7f9384009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:50.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.456+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9394160130 con 0x7f939415fd20 2024-05-27T06:50:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.648+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f939415a150 con 0x7f939415fd20 2024-05-27T06:50:50.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.648+0000 7f938aa00640 1 -- 172.21.15.81:0/1346421228 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6074 (secure 0 0 0) 0x7f937c063430 con 0x7f939415fd20 2024-05-27T06:50:50.644 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:50.644 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:48.327819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:50:50.644 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 30 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 msgr2=0x7f9370071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 0x7f9370071910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9384009ac0 tx=0x7f9384009190 comp rx=0 tx=0).stop 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 msgr2=0x7f93940c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c3c0 tx=0x7f937c00c880 comp rx=0 tx=0).stop 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 shutdown_connections 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f937006f400 0x7f9370071910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93940bf2b0 0x7f93940c0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f939415fd20 0x7f93940c6780 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 --2- 172.21.15.81:0/1346421228 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f93940c89a0 0x7f93941f6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 >> 172.21.15.81:0/1346421228 conn(0x7f939406f230 msgr2=0x7f93940c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 shutdown_connections 2024-05-27T06:50:50.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:50.652+0000 7f9398a00640 1 -- 172.21.15.81:0/1346421228 wait complete. 2024-05-27T06:50:50.666 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:50:48.327819+0000", "qdb_cluster": [ 4285, 4870, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './276/37' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './276' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/21' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/27' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/31' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/36' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/20' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/39' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/34' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/38' 2024-05-27T06:50:50.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/28' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/24' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/29' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/32' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/22' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/25' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/30' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/33' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/23' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/35' 2024-05-27T06:50:50.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/26' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './277/37' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './277' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/21' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/27' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/31' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/36' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/20' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/39' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/34' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/38' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/28' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/24' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/29' 2024-05-27T06:50:50.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/32' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/22' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/25' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/30' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/33' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/23' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/35' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/26' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './278/37' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './278' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/21' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/27' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/31' 2024-05-27T06:50:50.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/36' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/20' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/39' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/34' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/38' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/28' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/24' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/29' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/32' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/22' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/25' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/30' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/33' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/23' 2024-05-27T06:50:50.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/35' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/26' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './279/37' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './279' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/21' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/27' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/31' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/36' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/20' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/39' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/34' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/38' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/28' 2024-05-27T06:50:50.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/24' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/29' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/32' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/22' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/25' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/30' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/33' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/23' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/35' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/26' 2024-05-27T06:50:50.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './280/37' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './280' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/21' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/27' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/31' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/36' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/20' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/39' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/34' 2024-05-27T06:50:50.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/38' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/28' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/24' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/29' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/32' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/22' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/25' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/30' 2024-05-27T06:50:50.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/33' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/23' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/35' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/26' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './281/37' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './281' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/21' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/27' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/31' 2024-05-27T06:50:50.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/36' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/20' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/39' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/34' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/38' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/28' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/24' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/29' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/32' 2024-05-27T06:50:50.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/22' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/25' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/30' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/33' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/23' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/35' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/26' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed './282/37' 2024-05-27T06:50:50.774 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './282' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/21' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/27' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/31' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/36' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/20' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/39' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/34' 2024-05-27T06:50:50.775 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/38' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/28' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/24' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/29' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/32' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/22' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/25' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/30' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/33' 2024-05-27T06:50:50.776 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/23' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/35' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/26' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './283/37' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './283' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/21' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/27' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/31' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/36' 2024-05-27T06:50:50.777 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/20' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/39' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/34' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/38' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/28' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/24' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/29' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/32' 2024-05-27T06:50:50.778 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/22' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/25' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/30' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/33' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/23' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/35' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/26' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './284/37' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './284' 2024-05-27T06:50:50.779 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/21' 2024-05-27T06:50:50.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/27' 2024-05-27T06:50:50.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/31' 2024-05-27T06:50:50.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/36' 2024-05-27T06:50:50.780 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/20' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/39' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/34' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/38' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/28' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/24' 2024-05-27T06:50:50.781 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/29' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/32' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/22' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/25' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/30' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/33' 2024-05-27T06:50:50.782 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/23' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/35' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/26' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './285/37' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './285' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/21' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/27' 2024-05-27T06:50:50.783 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/31' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/36' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/20' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/39' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/34' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/38' 2024-05-27T06:50:50.784 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/28' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/24' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/29' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/32' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/22' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/25' 2024-05-27T06:50:50.785 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/30' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/33' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/23' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/35' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/26' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed './286/37' 2024-05-27T06:50:50.786 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './286' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/21' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/27' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/31' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/36' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/20' 2024-05-27T06:50:51.653 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/39' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/34' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/38' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/28' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/24' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/29' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/32' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/22' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/25' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/30' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/33' 2024-05-27T06:50:51.654 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/23' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/35' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/26' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './287/37' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './287' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/21' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/27' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/31' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/36' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/20' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/39' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/34' 2024-05-27T06:50:51.655 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/38' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/28' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/24' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/29' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/32' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/22' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/25' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/30' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/33' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/23' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/35' 2024-05-27T06:50:51.656 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/26' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './288/37' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './288' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/21' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/27' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/31' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/36' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/20' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/39' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/34' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/38' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/28' 2024-05-27T06:50:51.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/24' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/29' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/32' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/22' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/25' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/30' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/33' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/23' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/35' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/26' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './289/37' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './289' 2024-05-27T06:50:51.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/21' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/27' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/31' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/36' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/20' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/39' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/34' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/38' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/28' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/24' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/29' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/32' 2024-05-27T06:50:51.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/22' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/25' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/30' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/33' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/23' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/35' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/26' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './290/37' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './290' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/21' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/27' 2024-05-27T06:50:51.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/31' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/36' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/20' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/39' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/34' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/38' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/28' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/24' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/29' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/32' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/22' 2024-05-27T06:50:51.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/25' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/30' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/33' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/23' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/35' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/26' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './291/37' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './291' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/21' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/27' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/31' 2024-05-27T06:50:51.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/36' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/20' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/39' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/34' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/38' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/28' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/24' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/29' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/32' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/22' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/25' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/30' 2024-05-27T06:50:51.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/33' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/23' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/35' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/26' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './292/37' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './292' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/21' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/27' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/31' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/36' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/20' 2024-05-27T06:50:51.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/39' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/34' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/38' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/28' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/24' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/29' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/32' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/22' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/25' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/30' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/33' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/23' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/35' 2024-05-27T06:50:51.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/26' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './293/37' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './293' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/21' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/27' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/31' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/36' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/20' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/39' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/34' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/38' 2024-05-27T06:50:51.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/28' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/24' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/29' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/32' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/22' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/25' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/30' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/33' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/23' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/35' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/26' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed './294/37' 2024-05-27T06:50:51.667 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './294' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/21' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/27' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/31' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/36' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/20' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/39' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/34' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/38' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/28' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/24' 2024-05-27T06:50:51.668 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/29' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/32' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/22' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/25' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/30' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/33' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/23' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/35' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/26' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './295/37' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './295' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/21' 2024-05-27T06:50:51.669 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/27' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/31' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/36' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/20' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/39' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/34' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/38' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/28' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/24' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/29' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/32' 2024-05-27T06:50:51.670 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/22' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/25' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/30' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/33' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/23' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/35' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/26' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './296/37' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './296' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/21' 2024-05-27T06:50:51.671 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/27' 2024-05-27T06:50:52.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:52.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.824+0000 7fd842a00640 1 Processor -- start 2024-05-27T06:50:52.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.824+0000 7fd842a00640 1 -- start start 2024-05-27T06:50:52.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd842a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd842a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 0x7fd83c167090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd842a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd83c15c710 con 0x7fd83c0ca2f0 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd842a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd83c06eb20 con 0x7fd83c0caca0 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd842a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd83c06f910 con 0x7fd83c0d02a0 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd833400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd833400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55476/0 (socket says 172.21.15.81:55476) 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd83be00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fd83c0ca2f0 0x7fd83c1606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:48454/0 (socket says 172.21.15.81:48454) 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd833400640 1 -- 172.21.15.81:0/3842153112 learned_addr learned my addr 172.21.15.81:0/3842153112 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:52.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd840a00640 1 --2- 172.21.15.81:0/3842153112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 0x7fd83c167090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd83b400640 1 -- 172.21.15.81:0/3842153112 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3178330070 0 0) 0x7fd83c15c710 con 0x7fd83c0ca2f0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd83b400640 1 -- 172.21.15.81:0/3842153112 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd820003530 con 0x7fd83c0ca2f0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd840a00640 1 -- 172.21.15.81:0/3842153112 >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 msgr2=0x7fd83c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd840a00640 1 --2- 172.21.15.81:0/3842153112 >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd840a00640 1 -- 172.21.15.81:0/3842153112 >> v1:172.21.15.106:6789/0 conn(0x7fd83c0ca2f0 legacy=0x7fd83c1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd840a00640 1 -- 172.21.15.81:0/3842153112 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83c06e560 con 0x7fd83c0d02a0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.828+0000 7fd833400640 1 --2- 172.21.15.81:0/3842153112 >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd840a00640 1 --2- 172.21.15.81:0/3842153112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 0x7fd83c167090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd83401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0799e933f76490 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd83b400640 1 -- 172.21.15.81:0/3842153112 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340b2070 con 0x7fd83c0d02a0 2024-05-27T06:50:52.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd83b400640 1 -- 172.21.15.81:0/3842153112 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8340a5940 con 0x7fd83c0d02a0 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd83b400640 1 -- 172.21.15.81:0/3842153112 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340a5d10 con 0x7fd83c0d02a0 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- 172.21.15.81:0/3842153112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 msgr2=0x7fd83c167090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 --2- 172.21.15.81:0/3842153112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 0x7fd83c167090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd83401c970 comp rx=0 tx=0).stop 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- 172.21.15.81:0/3842153112 shutdown_connections 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 --2- 172.21.15.81:0/3842153112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0d02a0 0x7fd83c167090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 --2- 172.21.15.81:0/3842153112 >> v2:172.21.15.81:3300/0 conn(0x7fd83c0caca0 0x7fd83c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- 172.21.15.81:0/3842153112 >> 172.21.15.81:0/3842153112 conn(0x7fd83c068ae0 msgr2=0x7fd83c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- 172.21.15.81:0/3842153112 shutdown_connections 2024-05-27T06:50:52.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- 172.21.15.81:0/3842153112 wait complete. 2024-05-27T06:50:52.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 Processor -- start 2024-05-27T06:50:52.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.832+0000 7fd842a00640 1 -- start start 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0ca2f0 0x7fd83c158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 0x7fd83c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c06e220 con 0x7fd83c0d02a0 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd83c15cf90 con 0x7fd83c0caca0 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd83c0cb0b0 con 0x7fd83c0ca2f0 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd833400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 0x7fd83c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd83be00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0ca2f0 0x7fd83c158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55480/0 (socket says 172.21.15.81:55480) 2024-05-27T06:50:52.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd833400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 0x7fd83c14fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:55086/0 (socket says 172.21.15.81:55086) 2024-05-27T06:50:52.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 -- 172.21.15.81:0/718052361 learned_addr learned my addr 172.21.15.81:0/718052361 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 -- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0ca2f0 msgr2=0x7fd83c158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0ca2f0 0x7fd83c158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 -- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 msgr2=0x7fd83c14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 0x7fd83c14fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 -- 172.21.15.81:0/718052361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83c1500e0 con 0x7fd83c0d02a0 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd840a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd834019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340b2050 con 0x7fd83c0d02a0 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8340be070 con 0x7fd83c0d02a0 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8340ad850 con 0x7fd83c0d02a0 2024-05-27T06:50:52.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83c1573d0 con 0x7fd83c0d02a0 2024-05-27T06:50:52.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.836+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83c15bd80 con 0x7fd83c0d02a0 2024-05-27T06:50:52.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd83c15bf50 con 0x7fd83c0d02a0 2024-05-27T06:50:52.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd8340ad9f0 con 0x7fd83c0d02a0 2024-05-27T06:50:52.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd838c00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 0x7fd818071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:52.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7fd83412a2c0 con 0x7fd83c0d02a0 2024-05-27T06:50:52.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd83be00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 0x7fd818071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:52.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.840+0000 7fd83be00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 0x7fd818071940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd82c007f60 tx=0x7fd82c007ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:52.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:52.844+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd834016560 con 0x7fd83c0d02a0 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/31' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/36' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/20' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/39' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/34' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/38' 2024-05-27T06:50:53.043 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/28' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/24' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/29' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/32' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/22' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/25' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/30' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/33' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/23' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/35' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/26' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './297/37' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './297' 2024-05-27T06:50:53.044 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/21' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/27' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/31' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/36' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/20' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/39' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/34' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/38' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/28' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/24' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/29' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/32' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/22' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/25' 2024-05-27T06:50:53.045 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/30' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/33' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/23' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/35' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/26' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './298/37' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './298' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/21' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/27' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/31' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/36' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/20' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/39' 2024-05-27T06:50:53.046 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/34' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/38' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/28' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/24' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/29' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/32' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/22' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/25' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/30' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/33' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/23' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/35' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/26' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed './299/37' 2024-05-27T06:50:53.047 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './299' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/19' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/15' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/20' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/12' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/10' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/11' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/16' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/14' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/18' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/13' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/166/17' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/166' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/19' 2024-05-27T06:50:53.048 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/15' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/20' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/12' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/10' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/11' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/16' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/14' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/18' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/13' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/147/17' 2024-05-27T06:50:53.049 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/147' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/19' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/15' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/20' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/12' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/10' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/11' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/16' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/14' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/18' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/13' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/151/17' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/151' 2024-05-27T06:50:53.050 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/19' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/15' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/20' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/12' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/10' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/11' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/16' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/14' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/18' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/13' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/170/17' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/170' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/19' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/15' 2024-05-27T06:50:53.051 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/20' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/12' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/10' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/11' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/16' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/14' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/18' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/13' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/167/17' 2024-05-27T06:50:53.052 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/167' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/19' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/15' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/20' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/12' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/10' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/11' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/16' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/14' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/18' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/13' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/140/17' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/140' 2024-05-27T06:50:53.053 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/19' 2024-05-27T06:50:53.054 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/15' 2024-05-27T06:50:53.054 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/20' 2024-05-27T06:50:53.054 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/12' 2024-05-27T06:50:53.054 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/10' 2024-05-27T06:50:53.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.064+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd83c06a560 con 0x7fd83c0d02a0 2024-05-27T06:50:53.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.064+0000 7fd838c00640 1 -- 172.21.15.81:0/718052361 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6074 (secure 0 0 0) 0x7fd8340f69d0 con 0x7fd83c0d02a0 2024-05-27T06:50:53.060 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:53.060 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:48.327819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:50:53.060 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 30 2024-05-27T06:50:53.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 msgr2=0x7fd818071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:53.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 0x7fd818071940 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd82c007f60 tx=0x7fd82c007ef0 comp rx=0 tx=0).stop 2024-05-27T06:50:53.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 msgr2=0x7fd83c0cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:53.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd834009d90 tx=0x7fd834019710 comp rx=0 tx=0).stop 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 shutdown_connections 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd81806f430 0x7fd818071940 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd83c0d02a0 0x7fd83c0cd9b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd83c0caca0 0x7fd83c14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 --2- 172.21.15.81:0/718052361 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd83c0ca2f0 0x7fd83c158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 >> 172.21.15.81:0/718052361 conn(0x7fd83c068ae0 msgr2=0x7fd83c15dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 shutdown_connections 2024-05-27T06:50:53.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:53.068+0000 7fd842a00640 1 -- 172.21.15.81:0/718052361 wait complete. 2024-05-27T06:50:53.089 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:50:53.474 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/11' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/16' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/14' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/18' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/13' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/142/17' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/142' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/19' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/15' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/20' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/12' 2024-05-27T06:50:53.475 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/10' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/11' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/16' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/14' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/18' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/13' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/150/17' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/150' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/19' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/15' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/20' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/12' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/10' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/11' 2024-05-27T06:50:53.476 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/16' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/14' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/18' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/13' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/105/17' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/105' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/19' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/15' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/20' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/12' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/10' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/11' 2024-05-27T06:50:53.477 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/16' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/14' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/18' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/13' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/149/17' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/149' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/19' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/15' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/20' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/12' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/10' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/11' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/16' 2024-05-27T06:50:53.478 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/14' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/18' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/13' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/109/17' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/109' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/19' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/15' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/20' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/12' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/10' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/11' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/16' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/14' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/18' 2024-05-27T06:50:53.479 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/13' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/158/17' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/158' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/19' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/15' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/20' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/12' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/10' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/11' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/16' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/14' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/18' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/13' 2024-05-27T06:50:53.480 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/106/17' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/106' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/19' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/15' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/20' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/12' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/10' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/11' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/16' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/14' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/18' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/13' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/189/17' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/189' 2024-05-27T06:50:53.481 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/19' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/15' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/20' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/12' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/10' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/11' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/16' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/14' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/18' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/13' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/133/17' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/133' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/19' 2024-05-27T06:50:53.482 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/15' 2024-05-27T06:50:53.483 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/20' 2024-05-27T06:50:53.483 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/12' 2024-05-27T06:50:53.483 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/10' 2024-05-27T06:50:53.483 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/11' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/16' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/14' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/18' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/13' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/173/17' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/173' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/19' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/15' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/20' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/12' 2024-05-27T06:50:53.835 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/10' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/11' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/16' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/14' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/18' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/13' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/174/17' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/174' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/19' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/15' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/20' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/12' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/10' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/11' 2024-05-27T06:50:53.836 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/16' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/14' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/18' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/13' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/155/17' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/155' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/19' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/15' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/20' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/12' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/10' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/11' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/16' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/14' 2024-05-27T06:50:53.837 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/18' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/13' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/114/17' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/114' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/19' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/15' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/20' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/12' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/10' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/11' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/16' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/14' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/18' 2024-05-27T06:50:53.838 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/13' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/159/17' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/159' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/19' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/15' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/20' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/12' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/10' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/11' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/16' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/14' 2024-05-27T06:50:53.839 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/18' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/13' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/187/17' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/187' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/19' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/15' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/20' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/12' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/10' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/11' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/16' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/14' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/18' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/13' 2024-05-27T06:50:53.840 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/185/17' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/185' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/19' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/15' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/20' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/12' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/10' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/11' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/16' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/14' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/18' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/13' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/120/17' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/120' 2024-05-27T06:50:53.841 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/19' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/15' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/20' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/12' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/10' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/11' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/16' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/14' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/18' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/13' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/110/17' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/110' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/19' 2024-05-27T06:50:53.842 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/15' 2024-05-27T06:50:53.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/20' 2024-05-27T06:50:53.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/12' 2024-05-27T06:50:53.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/10' 2024-05-27T06:50:53.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/11' 2024-05-27T06:50:53.843 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/16' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/14' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/18' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/13' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/137/17' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/137' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/19' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/15' 2024-05-27T06:50:54.614 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/20' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/12' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/10' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/11' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/16' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/14' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/18' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/13' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/198/17' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/198' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/19' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/15' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/20' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/12' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/10' 2024-05-27T06:50:54.615 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/11' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/16' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/14' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/18' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/13' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/136/17' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/136' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/19' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/15' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/20' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/12' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/10' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/11' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/16' 2024-05-27T06:50:54.616 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/14' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/18' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/13' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/125/17' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/125' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/19' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/15' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/20' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/12' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/10' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/11' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/16' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/14' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/18' 2024-05-27T06:50:54.617 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/13' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/194/17' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/194' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/19' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/15' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/20' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/12' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/10' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/11' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/16' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/14' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/18' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/13' 2024-05-27T06:50:54.618 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/119/17' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/119' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/19' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/15' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/20' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/12' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/10' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/11' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/16' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/14' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/18' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/13' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/157/17' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/157' 2024-05-27T06:50:54.619 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/19' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/15' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/20' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/12' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/10' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/11' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/16' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/14' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/18' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/13' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/172/17' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/172' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/19' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/15' 2024-05-27T06:50:54.620 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/20' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/12' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/10' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/11' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/16' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/14' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/18' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/13' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/186/17' 2024-05-27T06:50:54.621 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/186' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/19' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/15' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/20' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/12' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/10' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/11' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/16' 2024-05-27T06:50:54.622 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/14' 2024-05-27T06:50:55.090 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:55.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 Processor -- start 2024-05-27T06:50:55.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 -- start start 2024-05-27T06:50:55.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9914156900 0x7f99140cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9914160470 0x7f9914160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99140cd390 con 0x7f991406c790 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f991415fd00 con 0x7f9914160470 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f991a600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f991415ffe0 con 0x7f9914156900 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9914160470 0x7f9914160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9914160470 0x7f9914160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55496/0 (socket says 172.21.15.81:55496) 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 -- 172.21.15.81:0/812440458 learned_addr learned my addr 172.21.15.81:0/812440458 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f9913400640 1 --2- 172.21.15.81:0/812440458 >> v2:172.21.15.106:3300/0 conn(0x7f9914156900 0x7f99140cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 -- 172.21.15.81:0/812440458 >> v2:172.21.15.106:3300/0 conn(0x7f9914156900 msgr2=0x7f99140cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 --2- 172.21.15.81:0/812440458 >> v2:172.21.15.106:3300/0 conn(0x7f9914156900 0x7f99140cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 -- 172.21.15.81:0/812440458 >> v1:172.21.15.106:6789/0 conn(0x7f991406c790 legacy=0x7f991415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.260+0000 7f990be00640 1 -- 172.21.15.81:0/812440458 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991406f120 con 0x7f9914160470 2024-05-27T06:50:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f990be00640 1 --2- 172.21.15.81:0/812440458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914160470 0x7f9914160880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9914072d90 tx=0x7f98fc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a967ba218c3dad24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:55.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f9912a00640 1 -- 172.21.15.81:0/812440458 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0b5070 con 0x7f9914160470 2024-05-27T06:50:55.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f9912a00640 1 -- 172.21.15.81:0/812440458 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc0a8a10 con 0x7f9914160470 2024-05-27T06:50:55.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f9912a00640 1 -- 172.21.15.81:0/812440458 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0a8de0 con 0x7f9914160470 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 -- 172.21.15.81:0/812440458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914160470 msgr2=0x7f9914160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 --2- 172.21.15.81:0/812440458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914160470 0x7f9914160880 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f9914072d90 tx=0x7f98fc01ed10 comp rx=0 tx=0).stop 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 -- 172.21.15.81:0/812440458 shutdown_connections 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 --2- 172.21.15.81:0/812440458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914160470 0x7f9914160880 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 --2- 172.21.15.81:0/812440458 >> v2:172.21.15.106:3300/0 conn(0x7f9914156900 0x7f99140cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 -- 172.21.15.81:0/812440458 >> 172.21.15.81:0/812440458 conn(0x7f99140cb2e0 msgr2=0x7f99140cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:55.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 -- 172.21.15.81:0/812440458 shutdown_connections 2024-05-27T06:50:55.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.264+0000 7f991a600640 1 -- 172.21.15.81:0/812440458 wait complete. 2024-05-27T06:50:55.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 Processor -- start 2024-05-27T06:50:55.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 -- start start 2024-05-27T06:50:55.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f991406c790 0x7f99141f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9914160470 0x7f99141f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f991406f9c0 con 0x7f9914156900 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f991406f6e0 con 0x7f991406c790 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f991a600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f991406d5b0 con 0x7f9914160470 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f990be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f9913400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f991406c790 0x7f99141f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f990be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55506/0 (socket says 172.21.15.81:55506) 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f990be00640 1 -- 172.21.15.81:0/2691535216 learned_addr learned my addr 172.21.15.81:0/2691535216 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f9913e00640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9914160470 0x7f99141f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.268+0000 7f990be00640 1 -- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9914160470 msgr2=0x7f99141f54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990be00640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9914160470 0x7f99141f54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990be00640 1 -- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f991406c790 msgr2=0x7f99141f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990be00640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f991406c790 0x7f99141f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990be00640 1 -- 172.21.15.81:0/2691535216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99141f3870 con 0x7f9914156900 2024-05-27T06:50:55.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990be00640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f98fc01b920 tx=0x7f98fc01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0be070 con 0x7f9914156900 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc004400 con 0x7f9914156900 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991415cfb0 con 0x7f9914156900 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0b5040 con 0x7f9914156900 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99140c9c70 con 0x7f9914156900 2024-05-27T06:50:55.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f991406a560 con 0x7f9914156900 2024-05-27T06:50:55.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f98fc0c1070 con 0x7f9914156900 2024-05-27T06:50:55.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.272+0000 7f990b400640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 0x7f98f0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:55.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.276+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f98fc12be00 con 0x7f9914156900 2024-05-27T06:50:55.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.276+0000 7f9913400640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 0x7f98f0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:55.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.276+0000 7f9913400640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 0x7f98f0071940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9904009ac0 tx=0x7f9904009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:55.272 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/18' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/13' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/190/17' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/190' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/19' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/15' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/20' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/12' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/10' 2024-05-27T06:50:55.273 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/11' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/16' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/14' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/18' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/13' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/104/17' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/104' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/19' 2024-05-27T06:50:55.274 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/15' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/20' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/12' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/10' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/11' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/16' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/14' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/18' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/13' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/164/17' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/164' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/19' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/15' 2024-05-27T06:50:55.275 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/20' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/12' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/10' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/11' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/16' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/14' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/18' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/13' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/112/17' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/112' 2024-05-27T06:50:55.276 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/19' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/15' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/20' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/12' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/10' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/11' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/16' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/14' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/18' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/13' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/100/17' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/100' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/19' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/15' 2024-05-27T06:50:55.277 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/20' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/12' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/10' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/11' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/16' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/14' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/18' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/13' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/134/17' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/134' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/19' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/15' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/20' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/12' 2024-05-27T06:50:55.278 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/10' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/11' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/16' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/14' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/18' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/13' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/115/17' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/115' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/19' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/15' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/20' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/12' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/10' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/11' 2024-05-27T06:50:55.279 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/16' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/14' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/18' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/13' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/116/17' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/116' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/19' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/15' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/20' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/12' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/10' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/11' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/16' 2024-05-27T06:50:55.280 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/14' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/18' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/13' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/195/17' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/195' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/19' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/15' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/20' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/12' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/10' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/11' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/16' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/14' 2024-05-27T06:50:55.281 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/18' 2024-05-27T06:50:55.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.280+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f98fc018870 con 0x7f9914156900 2024-05-27T06:50:55.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.476+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99140ccf40 con 0x7f9914156900 2024-05-27T06:50:55.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.476+0000 7f990b400640 1 -- 172.21.15.81:0/2691535216 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6076 (secure 0 0 0) 0x7f98fc0f85b0 con 0x7f9914156900 2024-05-27T06:50:55.473 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:55.473 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:53.338965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:resolve","state_seq":35,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:50:55.473 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 31 2024-05-27T06:50:55.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 msgr2=0x7f98f0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 0x7f98f0071940 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9904009ac0 tx=0x7f9904009190 comp rx=0 tx=0).stop 2024-05-27T06:50:55.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 msgr2=0x7f99140c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:55.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f98fc01b920 tx=0x7f98fc01bab0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 shutdown_connections 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9914160470 0x7f99141f54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f98f006f430 0x7f98f0071940 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9914156900 0x7f99140c76d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 --2- 172.21.15.81:0/2691535216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f991406c790 0x7f99141f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 >> 172.21.15.81:0/2691535216 conn(0x7f99140cb2e0 msgr2=0x7f99140c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 shutdown_connections 2024-05-27T06:50:55.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:55.480+0000 7f991a600640 1 -- 172.21.15.81:0/2691535216 wait complete. 2024-05-27T06:50:55.496 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:50:56.755 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/13' 2024-05-27T06:50:56.755 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/138/17' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/138' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/19' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/15' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/20' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/12' 2024-05-27T06:50:56.756 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/10' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/11' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/16' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/14' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/18' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/13' 2024-05-27T06:50:56.757 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/135/17' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/135' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/19' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/15' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/20' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/12' 2024-05-27T06:50:56.758 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/10' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/11' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/16' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/14' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/18' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/13' 2024-05-27T06:50:56.759 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/117/17' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/117' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/19' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/15' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/20' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/12' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/10' 2024-05-27T06:50:56.760 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/11' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/16' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/14' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/18' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/13' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/143/17' 2024-05-27T06:50:56.761 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/143' 2024-05-27T06:50:56.762 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/19' 2024-05-27T06:50:56.762 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/15' 2024-05-27T06:50:56.762 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/20' 2024-05-27T06:50:56.762 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/12' 2024-05-27T06:50:56.762 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/10' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/11' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/16' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/14' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/18' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/13' 2024-05-27T06:50:56.763 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/180/17' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/180' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/19' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/15' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/20' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/12' 2024-05-27T06:50:56.764 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/10' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/11' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/16' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/14' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/18' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/13' 2024-05-27T06:50:56.765 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/126/17' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/126' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/19' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/15' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/20' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/12' 2024-05-27T06:50:56.766 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/10' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/11' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/16' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/14' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/18' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/13' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/161/17' 2024-05-27T06:50:56.767 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/161' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/19' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/15' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/20' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/12' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/10' 2024-05-27T06:50:56.768 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/11' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/16' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/14' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/18' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/13' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/176/17' 2024-05-27T06:50:56.769 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/176' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/19' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/15' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/20' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/12' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/10' 2024-05-27T06:50:56.770 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/11' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/16' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/14' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/18' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/13' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/111/17' 2024-05-27T06:50:56.771 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/111' 2024-05-27T06:50:56.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/19' 2024-05-27T06:50:56.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/15' 2024-05-27T06:50:56.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/20' 2024-05-27T06:50:56.772 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/12' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/10' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/11' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/16' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/14' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/18' 2024-05-27T06:50:56.773 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/13' 2024-05-27T06:50:57.126 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/191/17' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/191' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/19' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/15' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/20' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/12' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/10' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/11' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/16' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/14' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/18' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/13' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/184/17' 2024-05-27T06:50:57.127 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/184' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/19' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/15' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/20' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/12' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/10' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/11' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/16' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/14' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/18' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/13' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/192/17' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/192' 2024-05-27T06:50:57.128 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/19' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/15' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/20' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/12' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/10' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/11' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/16' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/14' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/18' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/13' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/103/17' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/103' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/19' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/15' 2024-05-27T06:50:57.129 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/20' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/12' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/10' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/11' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/16' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/14' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/18' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/13' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/128/17' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/128' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/19' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/15' 2024-05-27T06:50:57.130 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/20' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/12' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/10' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/11' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/16' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/14' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/18' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/13' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/144/17' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/144' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/19' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/15' 2024-05-27T06:50:57.131 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/20' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/12' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/10' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/11' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/16' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/14' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/18' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/13' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/146/17' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/146' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/19' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/15' 2024-05-27T06:50:57.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/20' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/12' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/10' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/11' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/16' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/14' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/18' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/13' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/141/17' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/141' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/19' 2024-05-27T06:50:57.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/15' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/20' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/12' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/10' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/11' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/16' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/14' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/18' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/13' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/182/17' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/182' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/19' 2024-05-27T06:50:57.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/15' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/20' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/12' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/10' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/11' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/16' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/14' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/18' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/13' 2024-05-27T06:50:57.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/107/17' 2024-05-27T06:50:57.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:50:57.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 Processor -- start 2024-05-27T06:50:57.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 -- start start 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de40caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3de40cfdc0 0x7f3de40d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3de406e3e0 con 0x7f3de415d300 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f3de415ce70 con 0x7f3de40cfdc0 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3deb600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3de415d710 con 0x7f3de40ca2f0 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de3e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3de415d300 0x7f3de4160710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38648/0 (socket says 172.21.15.81:38648) 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de3e00640 1 -- 172.21.15.81:0/3706680731 learned_addr learned my addr 172.21.15.81:0/3706680731 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de9600640 1 --2- 172.21.15.81:0/3706680731 >> v2:172.21.15.106:3300/0 conn(0x7f3de40cfdc0 0x7f3de40d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de8c00640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de40caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de3400640 1 -- 172.21.15.81:0/3706680731 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867099213 0 0) 0x7f3de406e3e0 con 0x7f3de415d300 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de3400640 1 -- 172.21.15.81:0/3706680731 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dc8003530 con 0x7f3de415d300 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de9600640 1 -- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 msgr2=0x7f3de40caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de9600640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de40caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de9600640 1 -- 172.21.15.81:0/3706680731 >> v1:172.21.15.81:6789/0 conn(0x7f3de415d300 legacy=0x7f3de4160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.652+0000 7f3de9600640 1 -- 172.21.15.81:0/3706680731 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de406e6c0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3de9600640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de40cfdc0 0x7f3de40d01d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0097c0 tx=0x7f3ddc01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8793f364e65360cf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:57.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3de3400640 1 -- 172.21.15.81:0/3706680731 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc0b0070 con 0x7f3de40cfdc0 2024-05-27T06:50:57.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3de3400640 1 -- 172.21.15.81:0/3706680731 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ddc0ab600 con 0x7f3de40cfdc0 2024-05-27T06:50:57.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3de3400640 1 -- 172.21.15.81:0/3706680731 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc0ab9d0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3deb600640 1 -- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de40cfdc0 msgr2=0x7f3de40d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.656+0000 7f3deb600640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de40cfdc0 0x7f3de40d01d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc0097c0 tx=0x7f3ddc01c410 comp rx=0 tx=0).stop 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- 172.21.15.81:0/3706680731 shutdown_connections 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de40cfdc0 0x7f3de40d01d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 --2- 172.21.15.81:0/3706680731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de40caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- 172.21.15.81:0/3706680731 >> 172.21.15.81:0/3706680731 conn(0x7f3de4068ae0 msgr2=0x7f3de4068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- 172.21.15.81:0/3706680731 shutdown_connections 2024-05-27T06:50:57.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- 172.21.15.81:0/3706680731 wait complete. 2024-05-27T06:50:57.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 Processor -- start 2024-05-27T06:50:57.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- start start 2024-05-27T06:50:57.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de41f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de415d300 0x7f3de4154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3de406f880 con 0x7f3de40cfdc0 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3de406f700 con 0x7f3de415d300 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.660+0000 7f3deb600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3de41698b0 con 0x7f3de40ca2f0 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de41f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51140/0 (socket says 172.21.15.81:51140) 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 -- 172.21.15.81:0/4244573823 learned_addr learned my addr 172.21.15.81:0/4244573823 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:50:57.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de9600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de415d300 0x7f3de4154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 -- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 msgr2=0x7f3de41f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de41f97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 -- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de415d300 msgr2=0x7f3de4154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de415d300 0x7f3de4154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 -- 172.21.15.81:0/4244573823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de40cadc0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de3e00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c400 tx=0x7f3dd000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0017070 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dd0005cf0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de41f9ce0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd00072f0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de414a980 con 0x7f3de40cfdc0 2024-05-27T06:50:57.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.664+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de406a560 con 0x7f3de40cfdc0 2024-05-27T06:50:57.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.668+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3dd001a070 con 0x7f3de40cfdc0 2024-05-27T06:50:57.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.668+0000 7f3de0c00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 0x7f3dc4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:50:57.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.668+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f3dd008eb00 con 0x7f3de40cfdc0 2024-05-27T06:50:57.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.668+0000 7f3de8c00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 0x7f3dc4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:50:57.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.668+0000 7f3de8c00640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 0x7f3dc4071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4011340 tx=0x7f3dd400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:50:57.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.672+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3dd0021020 con 0x7f3de40cfdc0 2024-05-27T06:50:57.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3de40c9d10 con 0x7f3de40cfdc0 2024-05-27T06:50:57.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3de0c00640 1 -- 172.21.15.81:0/4244573823 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6076 (secure 0 0 0) 0x7f3de40cadc0 con 0x7f3de40cfdc0 2024-05-27T06:50:57.860 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:50:57.860 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:53.338965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:resolve","state_seq":35,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:50:57.860 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 31 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 msgr2=0x7f3dc4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 0x7f3dc4071910 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3dd4011340 tx=0x7f3dd400b040 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 msgr2=0x7f3de415bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.864+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000c400 tx=0x7f3dd000c8c0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 shutdown_connections 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3dc406f400 0x7f3dc4071910 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3de415d300 0x7f3de4154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de40cfdc0 0x7f3de415bc40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 --2- 172.21.15.81:0/4244573823 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3de40ca2f0 0x7f3de41f97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:50:57.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 >> 172.21.15.81:0/4244573823 conn(0x7f3de4068ae0 msgr2=0x7f3de41517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:50:57.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 shutdown_connections 2024-05-27T06:50:57.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:50:57.868+0000 7f3deb600640 1 -- 172.21.15.81:0/4244573823 wait complete. 2024-05-27T06:50:57.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:50:59.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/107' 2024-05-27T06:50:59.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/19' 2024-05-27T06:50:59.445 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/15' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/20' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/12' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/10' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/11' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/16' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/14' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/18' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/13' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/178/17' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/178' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/19' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/15' 2024-05-27T06:50:59.446 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/20' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/12' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/10' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/11' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/16' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/14' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/18' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/13' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/156/17' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/156' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/19' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/15' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/20' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/12' 2024-05-27T06:50:59.447 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/10' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/11' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/16' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/14' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/18' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/13' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/131/17' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/131' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/19' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/15' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/20' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/12' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/10' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/11' 2024-05-27T06:50:59.448 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/16' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/14' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/18' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/13' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/123/17' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/123' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/19' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/15' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/20' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/12' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/10' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/11' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/16' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/14' 2024-05-27T06:50:59.449 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/18' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/13' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/160/17' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/160' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/19' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/15' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/20' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/12' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/10' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/11' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/16' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/14' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/18' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/13' 2024-05-27T06:50:59.450 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/165/17' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/165' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/19' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/15' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/20' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/12' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/10' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/11' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/16' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/14' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/18' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/13' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/101/17' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/101' 2024-05-27T06:50:59.451 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/19' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/15' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/20' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/12' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/10' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/11' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/16' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/14' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/18' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/13' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/148/17' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/148' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/19' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/15' 2024-05-27T06:50:59.452 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/20' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/12' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/10' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/11' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/16' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/14' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/18' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/13' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/168/17' 2024-05-27T06:50:59.453 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/168' 2024-05-27T06:50:59.849 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/19' 2024-05-27T06:50:59.849 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/15' 2024-05-27T06:50:59.849 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/20' 2024-05-27T06:50:59.849 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/12' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/10' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/11' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/16' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/14' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/18' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/13' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/179/17' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/179' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/19' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/15' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/20' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/12' 2024-05-27T06:50:59.850 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/10' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/11' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/16' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/14' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/18' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/13' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/132/17' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/132' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/19' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/15' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/20' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/12' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/10' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/11' 2024-05-27T06:50:59.851 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/16' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/14' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/18' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/13' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/199/17' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/199' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/19' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/15' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/20' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/12' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/10' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/11' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/16' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/14' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/18' 2024-05-27T06:50:59.852 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/13' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/163/17' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/163' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/19' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/15' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/20' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/12' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/10' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/11' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/16' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/14' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/18' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/13' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/153/17' 2024-05-27T06:50:59.853 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/153' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/19' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/15' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/20' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/12' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/10' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/11' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/16' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/14' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/18' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/13' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/122/17' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/122' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/19' 2024-05-27T06:50:59.854 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/15' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/20' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/12' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/10' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/11' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/16' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/14' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/18' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/13' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/162/17' 2024-05-27T06:50:59.855 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/162' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/19' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/15' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/20' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/12' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/10' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/11' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/16' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/14' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/18' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/13' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/197/17' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/197' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/19' 2024-05-27T06:50:59.856 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/15' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/20' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/12' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/10' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/11' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/16' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/14' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/18' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/13' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/113/17' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/113' 2024-05-27T06:50:59.857 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/19' 2024-05-27T06:50:59.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:00.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 Processor -- start 2024-05-27T06:51:00.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 -- start start 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f40080ca770 0x7f40080cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f40080cbd00 0x7f40080cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40080c78b0 con 0x7f40080d02f0 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f40080c6ee0 con 0x7f40080ca770 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f4010200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f40080c71c0 con 0x7f40080cbd00 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f40080ca770 0x7f40080cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f40080ca770 0x7f40080cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51142/0 (socket says 172.21.15.81:51142) 2024-05-27T06:51:00.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 -- 172.21.15.81:0/1713912850 learned_addr learned my addr 172.21.15.81:0/1713912850 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:00.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400ce00640 1 --2- 172.21.15.81:0/1713912850 >> v2:172.21.15.106:3300/0 conn(0x7f40080cbd00 0x7f40080cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 -- 172.21.15.81:0/1713912850 >> v2:172.21.15.106:3300/0 conn(0x7f40080cbd00 msgr2=0x7f40080cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:00.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 --2- 172.21.15.81:0/1713912850 >> v2:172.21.15.106:3300/0 conn(0x7f40080cbd00 0x7f40080cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 -- 172.21.15.81:0/1713912850 >> v1:172.21.15.81:6789/0 conn(0x7f40080d02f0 legacy=0x7f4008154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.068+0000 7f400d800640 1 -- 172.21.15.81:0/1713912850 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40080c7b70 con 0x7f40080ca770 2024-05-27T06:51:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f400d800640 1 --2- 172.21.15.81:0/1713912850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080ca770 0x7f40080cb7c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4008073180 tx=0x7f3ffc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf4ed43d4392042 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:00.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4007e00640 1 -- 172.21.15.81:0/1713912850 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0ab430 con 0x7f40080ca770 2024-05-27T06:51:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4007e00640 1 -- 172.21.15.81:0/1713912850 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ffc0b0070 con 0x7f40080ca770 2024-05-27T06:51:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4007e00640 1 -- 172.21.15.81:0/1713912850 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc0ab7f0 con 0x7f40080ca770 2024-05-27T06:51:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 -- 172.21.15.81:0/1713912850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080ca770 msgr2=0x7f40080cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 --2- 172.21.15.81:0/1713912850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080ca770 0x7f40080cb7c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4008073180 tx=0x7f3ffc01c520 comp rx=0 tx=0).stop 2024-05-27T06:51:00.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 -- 172.21.15.81:0/1713912850 shutdown_connections 2024-05-27T06:51:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 --2- 172.21.15.81:0/1713912850 >> v2:172.21.15.106:3300/0 conn(0x7f40080cbd00 0x7f40080cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 --2- 172.21.15.81:0/1713912850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080ca770 0x7f40080cb7c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 -- 172.21.15.81:0/1713912850 >> 172.21.15.81:0/1713912850 conn(0x7f4008068ae0 msgr2=0x7f4008068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 -- 172.21.15.81:0/1713912850 shutdown_connections 2024-05-27T06:51:00.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.072+0000 7f4010200640 1 -- 172.21.15.81:0/1713912850 wait complete. 2024-05-27T06:51:00.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 Processor -- start 2024-05-27T06:51:00.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 -- start start 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40080ca770 0x7f40081488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40080cbd00 0x7f400814af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4008167590 con 0x7f40080d02f0 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4008154b60 con 0x7f40080cbd00 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f4010200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4008167710 con 0x7f40080ca770 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51148/0 (socket says 172.21.15.81:51148) 2024-05-27T06:51:00.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 -- 172.21.15.81:0/1513601725 learned_addr learned my addr 172.21.15.81:0/1513601725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400d800640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40080ca770 0x7f40081488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400ce00640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40080cbd00 0x7f400814af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 -- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40080ca770 msgr2=0x7f40081488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40080ca770 0x7f40081488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 -- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40080cbd00 msgr2=0x7f400814af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40080cbd00 0x7f400814af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 -- 172.21.15.81:0/1513601725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4008148df0 con 0x7f40080d02f0 2024-05-27T06:51:00.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.076+0000 7f400e200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f400000c710 tx=0x7f400000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000003b80 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4000003d20 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400806f880 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400000aa90 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40081f62c0 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f400806a560 con 0x7f40080d02f0 2024-05-27T06:51:00.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f400000ac30 con 0x7f40080d02f0 2024-05-27T06:51:00.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 0x7f3fe8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:00.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f400008f0e0 con 0x7f40080d02f0 2024-05-27T06:51:00.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.080+0000 7f400d800640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 0x7f3fe8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:00.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.084+0000 7f400d800640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 0x7f3fe8071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0af000 tx=0x7f3ffc0a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:00.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.088+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f400000ce20 con 0x7f40080d02f0 2024-05-27T06:51:00.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.280+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40081646c0 con 0x7f40080d02f0 2024-05-27T06:51:00.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.280+0000 7f4005600640 1 -- 172.21.15.81:0/1513601725 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6078 (secure 0 0 0) 0x7f400005b890 con 0x7f40080d02f0 2024-05-27T06:51:00.304 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:00.304 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:59.132406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:51:00.305 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 32 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 msgr2=0x7f3fe8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 0x7f3fe8071940 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0af000 tx=0x7f3ffc0a7000 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 msgr2=0x7f400814b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f400000c710 tx=0x7f400000cbd0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 shutdown_connections 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3fe806f430 0x7f3fe8071940 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40080d02f0 0x7f400814b490 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40080cbd00 0x7f400814af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 --2- 172.21.15.81:0/1513601725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40080ca770 0x7f40081488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 >> 172.21.15.81:0/1513601725 conn(0x7f4008068ae0 msgr2=0x7f4008068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 shutdown_connections 2024-05-27T06:51:00.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:00.284+0000 7f4010200640 1 -- 172.21.15.81:0/1513601725 wait complete. 2024-05-27T06:51:00.307 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:51:01.657 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/15' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/20' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/12' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/10' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/11' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/16' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/14' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/18' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/13' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/152/17' 2024-05-27T06:51:01.658 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/152' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/19' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/15' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/20' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/12' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/10' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/11' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/16' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/14' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/18' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/13' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/121/17' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/121' 2024-05-27T06:51:01.659 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/19' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/15' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/20' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/12' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/10' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/11' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/16' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/14' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/18' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/13' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/118/17' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/118' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/19' 2024-05-27T06:51:01.660 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/15' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/20' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/12' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/10' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/11' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/16' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/14' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/18' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/13' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/196/17' 2024-05-27T06:51:01.661 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/196' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/19' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/15' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/20' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/12' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/10' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/11' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/16' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/14' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/18' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/13' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/124/17' 2024-05-27T06:51:01.662 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/124' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/19' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/15' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/20' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/12' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/10' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/11' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/16' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/14' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/18' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/13' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/188/17' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/188' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/19' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/15' 2024-05-27T06:51:01.663 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/20' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/12' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/10' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/11' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/16' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/14' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/18' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/13' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/145/17' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/145' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/19' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/15' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/20' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/12' 2024-05-27T06:51:01.664 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/10' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/11' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/16' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/14' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/18' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/13' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/130/17' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/130' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/19' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/15' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/20' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/12' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/10' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/11' 2024-05-27T06:51:01.665 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/16' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/14' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/18' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/13' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/193/17' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/193' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/19' 2024-05-27T06:51:01.666 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/15' 2024-05-27T06:51:02.001 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/20' 2024-05-27T06:51:02.001 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/12' 2024-05-27T06:51:02.001 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/10' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/11' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/16' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/14' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/18' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/13' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/175/17' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/175' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/19' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/15' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/20' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/12' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/10' 2024-05-27T06:51:02.002 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/11' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/16' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/14' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/18' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/13' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/181/17' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/181' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/19' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/15' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/20' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/12' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/10' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/11' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/16' 2024-05-27T06:51:02.003 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/14' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/18' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/13' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/127/17' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/127' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/19' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/15' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/20' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/12' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/10' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/11' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/16' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/14' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/18' 2024-05-27T06:51:02.004 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/13' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/108/17' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/108' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/19' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/15' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/20' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/12' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/10' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/11' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/16' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/14' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/18' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/13' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/183/17' 2024-05-27T06:51:02.005 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/183' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/19' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/15' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/20' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/12' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/10' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/11' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/16' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/14' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/18' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/13' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/177/17' 2024-05-27T06:51:02.006 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/177' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/19' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/15' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/20' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/12' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/10' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/11' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/16' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/14' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/18' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/13' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/169/17' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/169' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/19' 2024-05-27T06:51:02.007 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/15' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/20' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/12' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/10' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/11' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/16' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/14' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/18' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/13' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/154/17' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/154' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/19' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/15' 2024-05-27T06:51:02.008 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/20' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/12' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/10' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/11' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/16' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/14' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/18' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/13' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/171/17' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/171' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/19' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/15' 2024-05-27T06:51:02.009 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/20' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/12' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/10' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/11' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/16' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/14' 2024-05-27T06:51:02.132 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/18' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/13' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/139/17' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/139' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/19' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/15' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/20' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/12' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/10' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/11' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/16' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/14' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/18' 2024-05-27T06:51:02.133 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/13' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/102/17' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/102' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/19' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/15' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/20' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/12' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/10' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/11' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/16' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/14' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/18' 2024-05-27T06:51:02.134 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/13' 2024-05-27T06:51:02.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed './copyofsnap1/test1/129/17' 2024-05-27T06:51:02.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1/129' 2024-05-27T06:51:02.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1/test1' 2024-05-27T06:51:02.135 INFO:tasks.workunit.client.0.smithi081.stdout:removed directory './copyofsnap1' 2024-05-27T06:51:02.135 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:02.136 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:02.136 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:02.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:51:02.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:51:02.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:02.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 Processor -- start 2024-05-27T06:51:02.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 -- start start 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa498156bc0 0x7fa4980cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa4980cc830 0x7fa49815fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4980c99f0 con 0x7fa4981603b0 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa498156fd0 con 0x7fa498156bc0 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49f800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa4981607c0 con 0x7fa4980cc830 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa498156bc0 0x7fa4980cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa498156bc0 0x7fa4980cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51160/0 (socket says 172.21.15.81:51160) 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa49ce00640 1 -- 172.21.15.81:0/1724558285 learned_addr learned my addr 172.21.15.81:0/1724558285 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:02.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.460+0000 7fa497e00640 1 --2- 172.21.15.81:0/1724558285 >> v2:172.21.15.106:3300/0 conn(0x7fa4980cc830 0x7fa49815fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa49ce00640 1 -- 172.21.15.81:0/1724558285 >> v2:172.21.15.106:3300/0 conn(0x7fa4980cc830 msgr2=0x7fa49815fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa49ce00640 1 --2- 172.21.15.81:0/1724558285 >> v2:172.21.15.106:3300/0 conn(0x7fa4980cc830 0x7fa49815fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa49ce00640 1 -- 172.21.15.81:0/1724558285 >> v1:172.21.15.106:6789/0 conn(0x7fa4981603b0 legacy=0x7fa498071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa49ce00640 1 -- 172.21.15.81:0/1724558285 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49815c980 con 0x7fa498156bc0 2024-05-27T06:51:02.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa49ce00640 1 --2- 172.21.15.81:0/1724558285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa4980cc2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa4980737e0 tx=0x7fa48801f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409399243208f93e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:02.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa497400640 1 -- 172.21.15.81:0/1724558285 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880b4070 con 0x7fa498156bc0 2024-05-27T06:51:02.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa497400640 1 -- 172.21.15.81:0/1724558285 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4880af400 con 0x7fa498156bc0 2024-05-27T06:51:02.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.464+0000 7fa497400640 1 -- 172.21.15.81:0/1724558285 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4880af7d0 con 0x7fa498156bc0 2024-05-27T06:51:02.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- 172.21.15.81:0/1724558285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 msgr2=0x7fa4980cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- 172.21.15.81:0/1724558285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa4980cc2f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa4980737e0 tx=0x7fa48801f1e0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- 172.21.15.81:0/1724558285 shutdown_connections 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- 172.21.15.81:0/1724558285 >> v2:172.21.15.106:3300/0 conn(0x7fa4980cc830 0x7fa49815fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- 172.21.15.81:0/1724558285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa4980cc2f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- 172.21.15.81:0/1724558285 >> 172.21.15.81:0/1724558285 conn(0x7fa498068cb0 msgr2=0x7fa4980745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- 172.21.15.81:0/1724558285 shutdown_connections 2024-05-27T06:51:02.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- 172.21.15.81:0/1724558285 wait complete. 2024-05-27T06:51:02.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 Processor -- start 2024-05-27T06:51:02.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- start start 2024-05-27T06:51:02.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 0x7fa49815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4981603b0 0x7fa498145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa49815d220 con 0x7fa498156bc0 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa49815cf40 con 0x7fa4980cc830 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.468+0000 7fa49f800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa49815d500 con 0x7fa4981603b0 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51162/0 (socket says 172.21.15.81:51162) 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 -- 172.21.15.81:0/2202213807 learned_addr learned my addr 172.21.15.81:0/2202213807 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49ce00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 0x7fa49815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49d800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4981603b0 0x7fa498145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 -- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4981603b0 msgr2=0x7fa498145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4981603b0 0x7fa498145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 -- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 msgr2=0x7fa49815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 0x7fa49815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 -- 172.21.15.81:0/2202213807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49815be90 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49ce00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 0x7fa49815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa497e00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e8a0 tx=0x7fa48000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480017070 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48000ee90 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000cb40 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49815dae0 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49815f7b0 con 0x7fa498156bc0 2024-05-27T06:51:02.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa49806a560 con 0x7fa498156bc0 2024-05-27T06:51:02.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.472+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa48001a070 con 0x7fa498156bc0 2024-05-27T06:51:02.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.476+0000 7fa494c00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 0x7fa474071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:02.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.476+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7fa48008f620 con 0x7fa498156bc0 2024-05-27T06:51:02.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.476+0000 7fa49ce00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 0x7fa474071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:02.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.476+0000 7fa49ce00640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 0x7fa474071910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4880b1000 tx=0x7fa4880aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:02.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.480+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa48005bca0 con 0x7fa498156bc0 2024-05-27T06:51:02.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa49814d830 con 0x7fa498156bc0 2024-05-27T06:51:02.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa494c00640 1 -- 172.21.15.81:0/2202213807 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6078 (secure 0 0 0) 0x7fa48005ffa0 con 0x7fa498156bc0 2024-05-27T06:51:02.667 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:02.667 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:50:59.132406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302]},"id":1}]} 2024-05-27T06:51:02.667 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 32 2024-05-27T06:51:02.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 msgr2=0x7fa474071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 0x7fa474071910 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa4880b1000 tx=0x7fa4880aa000 comp rx=0 tx=0).stop 2024-05-27T06:51:02.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 msgr2=0x7fa49814f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa48000e8a0 tx=0x7fa48000ed60 comp rx=0 tx=0).stop 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 shutdown_connections 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa4981603b0 0x7fa498145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa47406f400 0x7fa474071910 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa498156bc0 0x7fa49814f190 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 --2- 172.21.15.81:0/2202213807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4980cc830 0x7fa49815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 >> 172.21.15.81:0/2202213807 conn(0x7fa498068cb0 msgr2=0x7fa498071d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 shutdown_connections 2024-05-27T06:51:02.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:02.672+0000 7fa49f800640 1 -- 172.21.15.81:0/2202213807 wait complete. 2024-05-27T06:51:02.689 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4285": { "addr": "172.21.15.106:6837/13170464", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 13170464, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 13170464, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4285, "incarnation": 12, "join_fscid": -1, "name": "b", "rank": 2, "state": "up:active", "state_seq": 8 }, "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:50:59.132406+0000", "qdb_cluster": [ 4285, 4870, 4287, 4302 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 4285, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:51:04.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:04.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 Processor -- start 2024-05-27T06:51:04.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 -- start start 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74a815fd20 0x7f74a8156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a8072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a8160130 con 0x7f74a80ca900 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f74a80c9070 con 0x7f74a815fd20 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74ace00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f74a8074890 con 0x7f74a80c89a0 2024-05-27T06:51:04.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74a815fd20 0x7f74a8156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74a815fd20 0x7f74a8156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51170/0 (socket says 172.21.15.81:51170) 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 -- 172.21.15.81:0/2105938313 learned_addr learned my addr 172.21.15.81:0/2105938313 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f749fe00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a8072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 -- 172.21.15.81:0/2105938313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 msgr2=0x7f74a8072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a8072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 -- 172.21.15.81:0/2105938313 >> v1:172.21.15.106:6789/0 conn(0x7f74a80ca900 legacy=0x7f74a80cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:04.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.856+0000 7f74a5e00640 1 -- 172.21.15.81:0/2105938313 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a80c94c0 con 0x7f74a815fd20 2024-05-27T06:51:04.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74a5e00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8156a30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7498009870 tx=0x7f749801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6d2090fb370044 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:04.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74a5400640 1 -- 172.21.15.81:0/2105938313 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74980b0070 con 0x7f74a815fd20 2024-05-27T06:51:04.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74a5400640 1 -- 172.21.15.81:0/2105938313 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74980ab600 con 0x7f74a815fd20 2024-05-27T06:51:04.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74a5400640 1 -- 172.21.15.81:0/2105938313 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74980ab9d0 con 0x7f74a815fd20 2024-05-27T06:51:04.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 -- 172.21.15.81:0/2105938313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 msgr2=0x7f74a8156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:04.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8156a30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7498009870 tx=0x7f749801c560 comp rx=0 tx=0).stop 2024-05-27T06:51:04.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 -- 172.21.15.81:0/2105938313 shutdown_connections 2024-05-27T06:51:04.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a8072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:04.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 --2- 172.21.15.81:0/2105938313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8156a30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:04.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 -- 172.21.15.81:0/2105938313 >> 172.21.15.81:0/2105938313 conn(0x7f74a806f230 msgr2=0x7f74a8070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:04.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 -- 172.21.15.81:0/2105938313 shutdown_connections 2024-05-27T06:51:04.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.860+0000 7f74ace00640 1 -- 172.21.15.81:0/2105938313 wait complete. 2024-05-27T06:51:04.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 Processor -- start 2024-05-27T06:51:04.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 -- start start 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a81f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74a80ca900 0x7f74a80ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74a80c9760 con 0x7f74a815fd20 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8073fe0 con 0x7f74a80ca900 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f74a81698d0 con 0x7f74a80c89a0 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51180/0 (socket says 172.21.15.81:51180) 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 -- 172.21.15.81:0/2160409549 learned_addr learned my addr 172.21.15.81:0/2160409549 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f749fe00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74a80ca900 0x7f74a80ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a5e00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a81f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 -- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 msgr2=0x7f74a81f9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a81f9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 -- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74a80ca900 msgr2=0x7f74a80ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74a80ca900 0x7f74a80ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 -- 172.21.15.81:0/2160409549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a81f9f90 con 0x7f74a815fd20 2024-05-27T06:51:04.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74a6800640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c710 tx=0x7f74a000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0003b80 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a0003d20 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a000aa30 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.864+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a8154280 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a8166b60 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a806a560 con 0x7f74a815fd20 2024-05-27T06:51:04.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f74a000abd0 con 0x7f74a815fd20 2024-05-27T06:51:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f749e000640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 0x7f7484071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f74a008f1b0 con 0x7f74a815fd20 2024-05-27T06:51:04.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.868+0000 7f74a5e00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 0x7f7484071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:04.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.872+0000 7f74a5e00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 0x7f7484071940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74980ad000 tx=0x7f74980a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:04.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:04.876+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f74a8160130 con 0x7f74a815fd20 2024-05-27T06:51:05.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.064+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74a815a150 con 0x7f74a815fd20 2024-05-27T06:51:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.064+0000 7f749e000640 1 -- 172.21.15.81:0/2160409549 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6075 (secure 0 0 0) 0x7f74a000ce20 con 0x7f74a815fd20 2024-05-27T06:51:05.061 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:05.061 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:04.656754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:51:05.061 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 33 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 msgr2=0x7f7484071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 0x7f7484071940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74980ad000 tx=0x7f74980a6000 comp rx=0 tx=0).stop 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 msgr2=0x7f74a8145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c710 tx=0x7f74a000cbd0 comp rx=0 tx=0).stop 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 shutdown_connections 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f748406f430 0x7f7484071940 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a815fd20 0x7f74a8145980 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:05.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74a80ca900 0x7f74a80ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:05.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 --2- 172.21.15.81:0/2160409549 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f74a80c89a0 0x7f74a81f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:05.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 >> 172.21.15.81:0/2160409549 conn(0x7f74a806f230 msgr2=0x7f74a80cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:05.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 shutdown_connections 2024-05-27T06:51:05.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:05.068+0000 7f74ace00640 1 -- 172.21.15.81:0/2160409549 wait complete. 2024-05-27T06:51:05.082 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:51:07.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:07.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.248+0000 7f0bebe00640 1 Processor -- start 2024-05-27T06:51:07.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.248+0000 7f0bebe00640 1 -- start start 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0bebe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0be415ac20 0x7f0be40c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0bebe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be415db00 con 0x7f0be40c6fd0 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0bebe00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be4073860 con 0x7f0be41546f0 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0bebe00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0be415e6f0 con 0x7f0be415ac20 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be8a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0be40c6fd0 0x7f0be415a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35862/0 (socket says 172.21.15.81:35862) 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be8a00640 1 -- 172.21.15.81:0/2293769187 learned_addr learned my addr 172.21.15.81:0/2293769187 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:51:07.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be9e00640 1 --2- 172.21.15.81:0/2293769187 >> v2:172.21.15.106:3300/0 conn(0x7f0be415ac20 0x7f0be40c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 948730155 0 0) 0x7f0be415db00 con 0x7f0be40c6fd0 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bcc003530 con 0x7f0be40c6fd0 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be9e00640 1 -- 172.21.15.81:0/2293769187 >> v1:172.21.15.106:6789/0 conn(0x7f0be41546f0 legacy=0x7f0be4072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be9e00640 1 -- 172.21.15.81:0/2293769187 >> v1:172.21.15.81:6789/0 conn(0x7f0be40c6fd0 legacy=0x7f0be415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1630836270 0 0) 0x7f0bcc003530 con 0x7f0be40c6fd0 2024-05-27T06:51:07.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.252+0000 7f0be9e00640 1 -- 172.21.15.81:0/2293769187 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be415ddc0 con 0x7f0be415ac20 2024-05-27T06:51:07.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0be9e00640 1 --2- 172.21.15.81:0/2293769187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be415ac20 0x7f0be40c73e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009d90 tx=0x7f0bdc01f080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c7757fa6cbe3cd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:07.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc0b5070 con 0x7f0be415ac20 2024-05-27T06:51:07.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bdc0a8c10 con 0x7f0be415ac20 2024-05-27T06:51:07.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0be3e00640 1 -- 172.21.15.81:0/2293769187 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0001700 con 0x7f0be415ac20 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 -- 172.21.15.81:0/2293769187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be415ac20 msgr2=0x7f0be40c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 --2- 172.21.15.81:0/2293769187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be415ac20 0x7f0be40c73e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009d90 tx=0x7f0bdc01f080 comp rx=0 tx=0).stop 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 -- 172.21.15.81:0/2293769187 shutdown_connections 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 --2- 172.21.15.81:0/2293769187 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be415ac20 0x7f0be40c73e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 -- 172.21.15.81:0/2293769187 >> 172.21.15.81:0/2293769187 conn(0x7f0be4070700 msgr2=0x7f0be40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 -- 172.21.15.81:0/2293769187 shutdown_connections 2024-05-27T06:51:07.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 -- 172.21.15.81:0/2293769187 wait complete. 2024-05-27T06:51:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.256+0000 7f0bebe00640 1 Processor -- start 2024-05-27T06:51:07.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- start start 2024-05-27T06:51:07.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:07.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be41546f0 0x7f0be414f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:07.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0be415ac20 0x7f0be413fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0be415e380 con 0x7f0be40c6fd0 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0be40755b0 con 0x7f0be41546f0 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0be4154b00 con 0x7f0be415ac20 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40506/0 (socket says 172.21.15.81:40506) 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 -- 172.21.15.81:0/156688526 learned_addr learned my addr 172.21.15.81:0/156688526 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9e00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0be415ac20 0x7f0be413fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be8a00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be41546f0 0x7f0be414f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 -- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0be415ac20 msgr2=0x7f0be413fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0be415ac20 0x7f0be413fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 -- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be41546f0 msgr2=0x7f0be414f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be41546f0 0x7f0be414f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 -- 172.21.15.81:0/156688526 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be41f3490 con 0x7f0be40c6fd0 2024-05-27T06:51:07.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be9400640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800c450 tx=0x7f0bd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:07.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd80052d0 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd8005470 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be41f2a50 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8012850 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.260+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be415d360 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be406a560 con 0x7f0be40c6fd0 2024-05-27T06:51:07.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0bd80129f0 con 0x7f0be40c6fd0 2024-05-27T06:51:07.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0be1600640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 0x7f0bc4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:07.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7f0bd808f0d0 con 0x7f0be40c6fd0 2024-05-27T06:51:07.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0be8a00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 0x7f0bc4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:07.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.264+0000 7f0be8a00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 0x7f0bc4071910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0be414ec30 tx=0x7f0bd0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:07.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.268+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0bd805be30 con 0x7f0be40c6fd0 2024-05-27T06:51:07.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0be4076530 con 0x7f0be40c6fd0 2024-05-27T06:51:07.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0be1600640 1 -- 172.21.15.81:0/156688526 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6075 (secure 0 0 0) 0x7f0bd805b730 con 0x7f0be40c6fd0 2024-05-27T06:51:07.467 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:07.467 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:04.656754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:51:07.467 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 33 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 msgr2=0x7f0bc4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 0x7f0bc4071910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0be414ec30 tx=0x7f0bd0009040 comp rx=0 tx=0).stop 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 msgr2=0x7f0be41f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800c450 tx=0x7f0bd800c910 comp rx=0 tx=0).stop 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 shutdown_connections 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0be415ac20 0x7f0be413fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0bc406f400 0x7f0bc4071910 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0be41546f0 0x7f0be414f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 --2- 172.21.15.81:0/156688526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be40c6fd0 0x7f0be41f2f50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:07.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 >> 172.21.15.81:0/156688526 conn(0x7f0be4070700 msgr2=0x7f0be40ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:07.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 shutdown_connections 2024-05-27T06:51:07.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:07.472+0000 7f0bebe00640 1 -- 172.21.15.81:0/156688526 wait complete. 2024-05-27T06:51:07.489 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:51:09.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:09.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 Processor -- start 2024-05-27T06:51:09.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 -- start start 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb1240cc8c0 0x7fb1240cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1240caf50 con 0x7fb1241606c0 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12415c850 con 0x7fb12406e290 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb12aa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb12406e740 con 0x7fb1240cc8c0 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb128a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb1240cc8c0 0x7fb1240cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:09.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb11b400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb1241606c0 0x7fb1240ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35870/0 (socket says 172.21.15.81:35870) 2024-05-27T06:51:09.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb11b400640 1 -- 172.21.15.81:0/608290144 learned_addr learned my addr 172.21.15.81:0/608290144 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:51:09.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb128a00640 1 -- 172.21.15.81:0/608290144 >> v1:172.21.15.106:6789/0 conn(0x7fb12406e290 legacy=0x7fb124156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb128a00640 1 -- 172.21.15.81:0/608290144 >> v1:172.21.15.81:6789/0 conn(0x7fb1241606c0 legacy=0x7fb1240ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb128a00640 1 -- 172.21.15.81:0/608290144 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12415caf0 con 0x7fb1240cc8c0 2024-05-27T06:51:09.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.660+0000 7fb123400640 1 -- 172.21.15.81:0/608290144 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192052158 0 0) 0x7fb1240caf50 con 0x7fb1241606c0 2024-05-27T06:51:09.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb128a00640 1 --2- 172.21.15.81:0/608290144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1240cc8c0 0x7fb1240cccd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009d90 tx=0x7fb11c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33982ecdc9f1a9c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:09.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb123400640 1 -- 172.21.15.81:0/608290144 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0b4070 con 0x7fb1240cc8c0 2024-05-27T06:51:09.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb123400640 1 -- 172.21.15.81:0/608290144 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb11c0af600 con 0x7fb1240cc8c0 2024-05-27T06:51:09.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb123400640 1 -- 172.21.15.81:0/608290144 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0af9d0 con 0x7fb1240cc8c0 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 -- 172.21.15.81:0/608290144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1240cc8c0 msgr2=0x7fb1240cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 --2- 172.21.15.81:0/608290144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1240cc8c0 0x7fb1240cccd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009d90 tx=0x7fb11c01f150 comp rx=0 tx=0).stop 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 -- 172.21.15.81:0/608290144 shutdown_connections 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 --2- 172.21.15.81:0/608290144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1240cc8c0 0x7fb1240cccd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 -- 172.21.15.81:0/608290144 >> 172.21.15.81:0/608290144 conn(0x7fb124068ab0 msgr2=0x7fb124068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 -- 172.21.15.81:0/608290144 shutdown_connections 2024-05-27T06:51:09.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.664+0000 7fb12aa00640 1 -- 172.21.15.81:0/608290144 wait complete. 2024-05-27T06:51:09.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 Processor -- start 2024-05-27T06:51:09.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 -- start start 2024-05-27T06:51:09.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb12406e290 0x7fb124150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:09.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1240cc8c0 0x7fb1240c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:09.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb12415d5d0 con 0x7fb1241606c0 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb12415d0b0 con 0x7fb1240cc8c0 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb12aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb124160ad0 con 0x7fb12406e290 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb128a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb128a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40522/0 (socket says 172.21.15.81:40522) 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb128a00640 1 -- 172.21.15.81:0/1576193976 learned_addr learned my addr 172.21.15.81:0/1576193976 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb11b400640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1240cc8c0 0x7fb1240c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb123e00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb12406e290 0x7fb124150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb128a00640 1 -- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb12406e290 msgr2=0x7fb124150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.668+0000 7fb128a00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb12406e290 0x7fb124150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb128a00640 1 -- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1240cc8c0 msgr2=0x7fb1240c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb128a00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1240cc8c0 0x7fb1240c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb128a00640 1 -- 172.21.15.81:0/1576193976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb124154c60 con 0x7fb1241606c0 2024-05-27T06:51:09.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb128a00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb11c01cd80 tx=0x7fb11c01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:09.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0b4040 con 0x7fb1241606c0 2024-05-27T06:51:09.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb11c0c1070 con 0x7fb1241606c0 2024-05-27T06:51:09.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1241f6a20 con 0x7fb1241606c0 2024-05-27T06:51:09.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0ba690 con 0x7fb1241606c0 2024-05-27T06:51:09.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1241f6fe0 con 0x7fb1241606c0 2024-05-27T06:51:09.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb124156ec0 con 0x7fb1241606c0 2024-05-27T06:51:09.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb11c0ba830 con 0x7fb1241606c0 2024-05-27T06:51:09.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 0x7fb100071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:09.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.672+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7fb11c12dfe0 con 0x7fb1241606c0 2024-05-27T06:51:09.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.676+0000 7fb123e00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 0x7fb100071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:09.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.676+0000 7fb123e00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 0x7fb100071910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb114000f70 tx=0x7fb114009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:09.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.680+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb11c018cb0 con 0x7fb1241606c0 2024-05-27T06:51:09.740 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-05-27T06:51:09.741 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-05-27T06:51:09.763 INFO:tasks.workunit.client.0.smithi081.stderr:+ set -e 2024-05-27T06:51:09.763 INFO:tasks.workunit.client.0.smithi081.stderr:+ touch foo 2024-05-27T06:51:09.767 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod +x foo 2024-05-27T06:51:09.768 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir .snap/s 2024-05-27T06:51:09.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ find .snap/s/foo -executable 2024-05-27T06:51:09.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep foo 2024-05-27T06:51:09.794 INFO:tasks.workunit.client.0.smithi081.stdout:.snap/s/foo 2024-05-27T06:51:09.794 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir .snap/s 2024-05-27T06:51:09.805 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm foo 2024-05-27T06:51:09.809 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:51:09.809 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:09.810 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:09.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:09.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1241549f0 con 0x7fb1241606c0 2024-05-27T06:51:09.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb120c00640 1 -- 172.21.15.81:0/1576193976 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6075 (secure 0 0 0) 0x7fb11c0b8e20 con 0x7fb1241606c0 2024-05-27T06:51:09.879 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:09.879 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:04.656754+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:51:09.879 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 33 2024-05-27T06:51:09.880 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-05-27T06:51:09.880 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 msgr2=0x7fb100071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 0x7fb100071910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb114000f70 tx=0x7fb114009040 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 msgr2=0x7fb1240c8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb11c01cd80 tx=0x7fb11c01bef0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 shutdown_connections 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb10006f400 0x7fb100071910 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1241606c0 0x7fb1240c8e10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1240cc8c0 0x7fb1240c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 --2- 172.21.15.81:0/1576193976 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb12406e290 0x7fb124150a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 >> 172.21.15.81:0/1576193976 conn(0x7fb124068ab0 msgr2=0x7fb124151c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:09.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 shutdown_connections 2024-05-27T06:51:09.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:09.884+0000 7fb12aa00640 1 -- 172.21.15.81:0/1576193976 wait complete. 2024-05-27T06:51:09.901 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:51:10.195 INFO:tasks.workunit.client.0.smithi081.stderr:+ set -e 2024-05-27T06:51:10.196 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir foo 2024-05-27T06:51:10.573 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo x 2024-05-27T06:51:10.573 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:11.389 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir foo/.snap/ss 2024-05-27T06:51:11.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:12.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 Processor -- start 2024-05-27T06:51:12.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 -- start start 2024-05-27T06:51:12.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efd00072430 0x7efd00159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd00072840 con 0x7efd0015a040 2024-05-27T06:51:12.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7efd000c5040 con 0x7efd00072430 2024-05-27T06:51:12.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efd04c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd000c75c0 con 0x7efd00161cd0 2024-05-27T06:51:12.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfe800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efd00072430 0x7efd00159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfe800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:46690/0 (socket says 172.21.15.81:46690) 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7efd00072430 0x7efd00159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:34734/0 (socket says 172.21.15.81:34734) 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfe800640 1 -- 172.21.15.81:0/4137373172 learned_addr learned my addr 172.21.15.81:0/4137373172 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcf5400640 1 --1- 172.21.15.81:0/4137373172 >> v1:172.21.15.106:6789/0 conn(0x7efd0015a040 0x7efd000c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:42178/0 (socket says 172.21.15.81:42178) 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 -- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 msgr2=0x7efd001640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 -- 172.21.15.81:0/4137373172 >> v1:172.21.15.106:6789/0 conn(0x7efd0015a040 legacy=0x7efd000c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfde00640 1 -- 172.21.15.81:0/4137373172 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd000c5270 con 0x7efd00072430 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfd400640 1 -- 172.21.15.81:0/4137373172 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141943993 0 0) 0x7efd00072840 con 0x7efd0015a040 2024-05-27T06:51:12.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.068+0000 7efcfe800640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:51:12.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efcfde00640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00072430 0x7efd00159b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcf00097c0 tx=0x7efcf001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df64c81125976db server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:12.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efcfd400640 1 -- 172.21.15.81:0/4137373172 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00b0070 con 0x7efd00072430 2024-05-27T06:51:12.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efcfd400640 1 -- 172.21.15.81:0/4137373172 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf00ab600 con 0x7efd00072430 2024-05-27T06:51:12.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efcfd400640 1 -- 172.21.15.81:0/4137373172 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00ab9d0 con 0x7efd00072430 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 -- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00072430 msgr2=0x7efd00159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00072430 0x7efd00159b00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcf00097c0 tx=0x7efcf001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 -- 172.21.15.81:0/4137373172 shutdown_connections 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00161cd0 0x7efd001640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 --2- 172.21.15.81:0/4137373172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00072430 0x7efd00159b00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 -- 172.21.15.81:0/4137373172 >> 172.21.15.81:0/4137373172 conn(0x7efd00070700 msgr2=0x7efd000cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 -- 172.21.15.81:0/4137373172 shutdown_connections 2024-05-27T06:51:12.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.072+0000 7efd04c00640 1 -- 172.21.15.81:0/4137373172 wait complete. 2024-05-27T06:51:12.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 Processor -- start 2024-05-27T06:51:12.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 -- start start 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00072430 0x7efd000cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00161cd0 0x7efd0014aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd000ce6f0 con 0x7efd0015a040 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd000c7dd0 con 0x7efd00161cd0 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efd04c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd000c5610 con 0x7efd00072430 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40538/0 (socket says 172.21.15.81:40538) 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 -- 172.21.15.81:0/3013629650 learned_addr learned my addr 172.21.15.81:0/3013629650 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcfde00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00072430 0x7efd000cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcfe800640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00161cd0 0x7efd0014aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 -- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00072430 msgr2=0x7efd000cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00072430 0x7efd000cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 -- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00161cd0 msgr2=0x7efd0014aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00161cd0 0x7efd0014aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.076+0000 7efcf5400640 1 -- 172.21.15.81:0/3013629650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0014bd60 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf5400640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efce800c430 tx=0x7efce800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce80074e0 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce80047f0 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0014c000 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8007680 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0014c5c0 con 0x7efd0015a040 2024-05-27T06:51:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0006a560 con 0x7efd0015a040 2024-05-27T06:51:12.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efce8011070 con 0x7efd0015a040 2024-05-27T06:51:12.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 0x7efcdc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:12.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.080+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6083+0+0 (secure 0 0 0) 0x7efce808f0c0 con 0x7efd0015a040 2024-05-27T06:51:12.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.084+0000 7efcfde00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 0x7efcdc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:12.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.084+0000 7efcfde00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 0x7efcdc071940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efcf00ad000 tx=0x7efcf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:12.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.088+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7efce805bd90 con 0x7efd0015a040 2024-05-27T06:51:12.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd000c5940 con 0x7efd0015a040 2024-05-27T06:51:12.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efcf6a00640 1 -- 172.21.15.81:0/3013629650 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6080 (secure 0 0 0) 0x7efce805b690 con 0x7efd0015a040 2024-05-27T06:51:12.274 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:12.274 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302,5049]},"id":1}]} 2024-05-27T06:51:12.275 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-05-27T06:51:12.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 msgr2=0x7efcdc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 0x7efcdc071940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efcf00ad000 tx=0x7efcf0008040 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 msgr2=0x7efd00149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efce800c430 tx=0x7efce800c8f0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 shutdown_connections 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efcdc06f430 0x7efcdc071940 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd00161cd0 0x7efd0014aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd0015a040 0x7efd00149ae0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 --2- 172.21.15.81:0/3013629650 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd00072430 0x7efd000cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 >> 172.21.15.81:0/3013629650 conn(0x7efd00070700 msgr2=0x7efd000c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 shutdown_connections 2024-05-27T06:51:12.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:12.280+0000 7efd04c00640 1 -- 172.21.15.81:0/3013629650 wait complete. 2024-05-27T06:51:12.296 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=3, replacing gid=4281 2024-05-27T06:51:12.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2024-05-27T06:51:15.655 INFO:tasks.workunit.client.0.smithi081.stderr:+ ln foo/x foo/xx 2024-05-27T06:51:15.660 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat foo/.snap/ss/x 2024-05-27T06:51:15.671 INFO:tasks.workunit.client.0.smithi081.stdout:x 2024-05-27T06:51:15.671 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir foo/.snap/ss 2024-05-27T06:51:15.688 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo a 2024-05-27T06:51:15.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo b 2024-05-27T06:51:15.691 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir foo/.snap/s 2024-05-27T06:51:16.484 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat foo/.snap/s/a 2024-05-27T06:51:16.497 INFO:tasks.workunit.client.0.smithi081.stderr:+ r=a 2024-05-27T06:51:16.497 INFO:tasks.workunit.client.0.smithi081.stderr:+ [ -z a ] 2024-05-27T06:51:16.498 INFO:tasks.workunit.client.0.smithi081.stderr:+ ln foo/b foo/b2 2024-05-27T06:51:16.502 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat foo/.snap/s/b 2024-05-27T06:51:16.512 INFO:tasks.workunit.client.0.smithi081.stdout:b 2024-05-27T06:51:16.513 INFO:tasks.workunit.client.0.smithi081.stdout:this used to hang: 2024-05-27T06:51:16.513 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo this used to hang: 2024-05-27T06:51:16.515 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo more 2024-05-27T06:51:16.517 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo oh, it didn't hang! good job. 2024-05-27T06:51:16.517 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat foo/b 2024-05-27T06:51:16.518 INFO:tasks.workunit.client.0.smithi081.stdout:oh, it didn't hang! good job. 2024-05-27T06:51:16.520 INFO:tasks.workunit.client.0.smithi081.stdout:b 2024-05-27T06:51:16.520 INFO:tasks.workunit.client.0.smithi081.stdout:more 2024-05-27T06:51:16.521 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir foo/.snap/s 2024-05-27T06:51:16.535 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm -r foo 2024-05-27T06:51:16.565 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:51:16.566 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:16.567 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:16.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:16.599 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-05-27T06:51:16.599 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-05-27T06:51:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ set -e 2024-05-27T06:51:16.667 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:16.672 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir .snap/bar 2024-05-27T06:51:16.685 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/foo 2024-05-27T06:51:16.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/foo 2024-05-27T06:51:16.858 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:16.862 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/bar/a 2024-05-27T06:51:16.869 INFO:tasks.workunit.client.0.smithi081.stdout: File: .snap/bar/a 2024-05-27T06:51:16.869 INFO:tasks.workunit.client.0.smithi081.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-05-27T06:51:16.869 INFO:tasks.workunit.client.0.smithi081.stdout:Device: 2ch/44d Inode: 845524441765765 Links: 2 2024-05-27T06:51:16.869 INFO:tasks.workunit.client.0.smithi081.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2024-05-27T06:51:16.870 INFO:tasks.workunit.client.0.smithi081.stdout:Access: 2024-05-27 06:51:16.679332229 +0000 2024-05-27T06:51:16.870 INFO:tasks.workunit.client.0.smithi081.stdout:Modify: 2024-05-27 06:51:16.679332229 +0000 2024-05-27T06:51:16.870 INFO:tasks.workunit.client.0.smithi081.stdout:Change: 2024-05-27 06:51:16.679332229 +0000 2024-05-27T06:51:16.870 INFO:tasks.workunit.client.0.smithi081.stdout: Birth: - 2024-05-27T06:51:16.870 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir .snap/bar 2024-05-27T06:51:16.949 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:51:16.949 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:16.950 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:16.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:17.021 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-05-27T06:51:17.022 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-05-27T06:51:17.085 INFO:tasks.workunit.client.0.smithi081.stderr:+ set -e 2024-05-27T06:51:17.085 INFO:tasks.workunit.client.0.smithi081.stderr:+ seq 1 20 2024-05-27T06:51:17.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:17.090 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:17.093 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:17.106 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:17.239 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:17.241 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:17.495 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:17.509 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:17.509 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:17.509 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:17.509 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:17.510 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:17.510 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:17.510 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:17.510 INFO:tasks.workunit.client.0.smithi081.stdout: 37582 ? 00:00:00 ps 2024-05-27T06:51:17.514 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:18.476 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:18.479 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:18.485 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:18.488 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:18.491 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:18.976 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:18.982 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:18.984 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:19.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:19.092 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:19.093 INFO:tasks.workunit.client.0.smithi081.stdout: 37592 ? 00:00:00 ps 2024-05-27T06:51:19.097 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:19.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:19.121 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:19.130 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:19.134 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:19.137 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:19.148 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:19.158 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:19.160 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:19.228 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:19.241 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:19.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37602 ? 00:00:00 ps 2024-05-27T06:51:19.246 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:19.268 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:19.272 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:19.278 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:19.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:19.283 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:19.294 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:19.298 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-05-27T06:51:19.298 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-27T06:51:19.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-27T06:51:19.301 INFO:tasks.ceph.mds.c:Started 2024-05-27T06:51:19.302 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:19.405 INFO:tasks.ceph.mds.c.smithi081.stdout:starting mds.c at 2024-05-27T06:51:19.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.460+0000 7f7f79000640 1 Processor -- start 2024-05-27T06:51:19.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.460+0000 7f7f79000640 1 -- start start 2024-05-27T06:51:19.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f79000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 0x7f7f74156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f79000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 0x7f7f740cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f79000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f740ca6f0 con 0x7f7f740d0380 2024-05-27T06:51:19.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f79000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7f740ca970 con 0x7f7f740ca2e0 2024-05-27T06:51:19.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f79000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7415c890 con 0x7f7f740c95c0 2024-05-27T06:51:19.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f72800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7f740d0380 0x7f7f7415fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:48822/0 (socket says 172.21.15.81:48822) 2024-05-27T06:51:19.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f72800640 1 -- 172.21.15.81:0/2036668718 learned_addr learned my addr 172.21.15.81:0/2036668718 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:51:19.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71e00640 1 --2- 172.21.15.81:0/2036668718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 0x7f7f74156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f69400640 1 --2- 172.21.15.81:0/2036668718 >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 0x7f7f740cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71400640 1 -- 172.21.15.81:0/2036668718 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1029430379 0 0) 0x7f7f6c0014a0 con 0x7f7f740d0380 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71400640 1 -- 172.21.15.81:0/2036668718 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f58003530 con 0x7f7f740d0380 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71e00640 1 -- 172.21.15.81:0/2036668718 >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 msgr2=0x7f7f740cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71e00640 1 --2- 172.21.15.81:0/2036668718 >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 0x7f7f740cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71e00640 1 -- 172.21.15.81:0/2036668718 >> v1:172.21.15.81:6789/0 conn(0x7f7f740d0380 legacy=0x7f7f7415fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f71e00640 1 -- 172.21.15.81:0/2036668718 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7415cb70 con 0x7f7f740c95c0 2024-05-27T06:51:19.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.464+0000 7f7f69400640 1 --2- 172.21.15.81:0/2036668718 >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 0x7f7f740cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:51:19.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f71e00640 1 --2- 172.21.15.81:0/2036668718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 0x7f7f74156b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097c0 tx=0x7f7f6401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d5029687e5dd805 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:19.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f71400640 1 -- 172.21.15.81:0/2036668718 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640b5070 con 0x7f7f740c95c0 2024-05-27T06:51:19.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f71400640 1 -- 172.21.15.81:0/2036668718 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f640a8990 con 0x7f7f740c95c0 2024-05-27T06:51:19.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f71400640 1 -- 172.21.15.81:0/2036668718 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f640a8d60 con 0x7f7f740c95c0 2024-05-27T06:51:19.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f79000640 1 -- 172.21.15.81:0/2036668718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 msgr2=0x7f7f74156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.468+0000 7f7f79000640 1 --2- 172.21.15.81:0/2036668718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 0x7f7f74156b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7f640097c0 tx=0x7f7f6401ec90 comp rx=0 tx=0).stop 2024-05-27T06:51:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 -- 172.21.15.81:0/2036668718 shutdown_connections 2024-05-27T06:51:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 --2- 172.21.15.81:0/2036668718 >> v2:172.21.15.106:3300/0 conn(0x7f7f740ca2e0 0x7f7f740cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 --2- 172.21.15.81:0/2036668718 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740c95c0 0x7f7f74156b00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 -- 172.21.15.81:0/2036668718 >> 172.21.15.81:0/2036668718 conn(0x7f7f74068ae0 msgr2=0x7f7f74068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:19.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 -- 172.21.15.81:0/2036668718 shutdown_connections 2024-05-27T06:51:19.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.472+0000 7f7f79000640 1 -- 172.21.15.81:0/2036668718 wait complete. 2024-05-27T06:51:19.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 Processor -- start 2024-05-27T06:51:19.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 -- start start 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f740c95c0 0x7f7f740c6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 0x7f7f741f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7415d410 con 0x7f7f740d0380 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7415d130 con 0x7f7f740c95c0 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f79000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7f74160470 con 0x7f7f740ca2e0 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f72800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 0x7f7f741f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f72800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48046/0 (socket says 172.21.15.81:48046) 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f72800640 1 -- 172.21.15.81:0/98926168 learned_addr learned my addr 172.21.15.81:0/98926168 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f71e00640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f740c95c0 0x7f7f740c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 -- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f740c95c0 msgr2=0x7f7f740c6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f740c95c0 0x7f7f740c6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 -- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 msgr2=0x7f7f741492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 -- 172.21.15.81:0/98926168 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f741517e0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f72800640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:51:19.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.476+0000 7f7f69400640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 0x7f7f741f3620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e8e0 tx=0x7f7f5c00eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:19.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c017070 con 0x7f7f740ca2e0 2024-05-27T06:51:19.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5c00c8e0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f741f3b60 con 0x7f7f740ca2e0 2024-05-27T06:51:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00ccd0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f74150eb0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7415f8a0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7f5c01a070 con 0x7f7f740ca2e0 2024-05-27T06:51:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.480+0000 7f7f6aa00640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 0x7f7f50071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.484+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(45..45 src has 1..45) ==== 6139+0+0 (secure 0 0 0) 0x7f7f5c08fad0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.484+0000 7f7f71e00640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 0x7f7f50071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:19.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.484+0000 7f7f71e00640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 0x7f7f50071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f640b2000 tx=0x7f7f64008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:19.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.484+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7f5c05c1a0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.525 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:19.527 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:19.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7406a560 con 0x7f7f740ca2e0 2024-05-27T06:51:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f6aa00640 1 -- 172.21.15.81:0/98926168 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6080 (secure 0 0 0) 0x7f7f5c0604a0 con 0x7f7f740ca2e0 2024-05-27T06:51:19.683 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:19.683 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4285,4870,4287,4302,5049]},"id":1}]} 2024-05-27T06:51:19.683 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 34 2024-05-27T06:51:19.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 msgr2=0x7f7f50071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 0x7f7f50071940 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7f640b2000 tx=0x7f7f64008040 comp rx=0 tx=0).stop 2024-05-27T06:51:19.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 msgr2=0x7f7f741f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:19.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 0x7f7f741f3620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00e8e0 tx=0x7f7f5c00eda0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 shutdown_connections 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7f740ca2e0 0x7f7f741f3620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7f5006f430 0x7f7f50071940 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f740d0380 0x7f7f741492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 --2- 172.21.15.81:0/98926168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7f740c95c0 0x7f7f740c6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 >> 172.21.15.81:0/98926168 conn(0x7f7f74068ae0 msgr2=0x7f7f7406df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 shutdown_connections 2024-05-27T06:51:19.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:19.688+0000 7f7f79000640 1 -- 172.21.15.81:0/98926168 wait complete. 2024-05-27T06:51:19.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-05-27T06:51:19.722 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:19.741 INFO:tasks.workunit.client.0.smithi081.stdout: 37660 ? 00:00:00 ps 2024-05-27T06:51:19.744 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:19.768 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:19.771 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:19.778 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:19.781 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:19.783 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:19.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:19.804 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:19.806 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:19.929 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:19.943 INFO:tasks.workunit.client.0.smithi081.stdout: 37670 ? 00:00:00 ps 2024-05-27T06:51:19.947 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:19.965 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:19.969 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:19.974 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:19.977 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:19.979 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:20.556 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:20.565 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:20.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:20.774 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:20.787 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:20.788 INFO:tasks.workunit.client.0.smithi081.stdout: 37681 ? 00:00:00 ps 2024-05-27T06:51:20.793 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:20.813 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:20.816 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:20.822 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:20.825 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:20.828 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:20.839 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:20.848 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:20.850 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:20.930 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:20.944 INFO:tasks.workunit.client.0.smithi081.stdout: 37691 ? 00:00:00 ps 2024-05-27T06:51:20.948 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:20.966 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:20.970 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:20.975 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:20.979 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:20.981 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:20.993 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:21.002 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:21.004 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:21.074 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:21.088 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:21.089 INFO:tasks.workunit.client.0.smithi081.stdout: 37701 ? 00:00:00 ps 2024-05-27T06:51:21.092 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:21.114 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:21.119 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:21.125 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:21.128 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:21.135 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:21.143 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:21.156 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:21.158 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:21.231 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:21.245 INFO:tasks.workunit.client.0.smithi081.stdout: 37711 ? 00:00:00 ps 2024-05-27T06:51:21.250 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:21.270 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:21.273 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:21.278 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:21.281 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:21.284 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:21.294 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:21.304 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:21.305 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:21.421 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:21.437 INFO:tasks.workunit.client.0.smithi081.stdout: 37721 ? 00:00:00 ps 2024-05-27T06:51:21.442 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:21.460 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:21.463 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:21.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:21.471 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:21.474 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:21.679 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:21.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:51:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.864+0000 7f8b09800640 1 Processor -- start 2024-05-27T06:51:21.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 -- start start 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b0406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b04162bd0 0x7f8b04164ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b040c6150 con 0x7f8b040c91c0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8b040c6410 con 0x7f8b04162bd0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b09800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040c6800 con 0x7f8b0406c790 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b03e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8b040c91c0 0x7f8b040caa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:48828/0 (socket says 172.21.15.81:48828) 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b03e00640 1 -- 172.21.15.81:0/283842252 learned_addr learned my addr 172.21.15.81:0/283842252 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b08e00640 1 --2- 172.21.15.81:0/283842252 >> v2:172.21.15.81:3300/0 conn(0x7f8b04162bd0 0x7f8b04164ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b03400640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b0406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b02a00640 1 -- 172.21.15.81:0/283842252 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040890206 0 0) 0x7f8b040c6150 con 0x7f8b040c91c0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b02a00640 1 -- 172.21.15.81:0/283842252 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ae8003530 con 0x7f8b040c91c0 2024-05-27T06:51:21.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b08e00640 1 -- 172.21.15.81:0/283842252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 msgr2=0x7f8b0406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:51:21.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b08e00640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b0406cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:21.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b08e00640 1 -- 172.21.15.81:0/283842252 >> v1:172.21.15.81:6789/0 conn(0x7f8b040c91c0 legacy=0x7f8b040caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:21.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.868+0000 7f8b08e00640 1 -- 172.21.15.81:0/283842252 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b040c6ae0 con 0x7f8b04162bd0 2024-05-27T06:51:21.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b08e00640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b04162bd0 0x7f8b04164ff0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009dc0 tx=0x7f8afc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659283065567a22a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:21.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b02a00640 1 -- 172.21.15.81:0/283842252 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0b2070 con 0x7f8b04162bd0 2024-05-27T06:51:21.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b02a00640 1 -- 172.21.15.81:0/283842252 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc0a5810 con 0x7f8b04162bd0 2024-05-27T06:51:21.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b02a00640 1 -- 172.21.15.81:0/283842252 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0a5be0 con 0x7f8b04162bd0 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 -- 172.21.15.81:0/283842252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b04162bd0 msgr2=0x7f8b04164ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b04162bd0 0x7f8b04164ff0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009dc0 tx=0x7f8afc01c950 comp rx=0 tx=0).stop 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 -- 172.21.15.81:0/283842252 shutdown_connections 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b04162bd0 0x7f8b04164ff0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 --2- 172.21.15.81:0/283842252 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b0406cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:21.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 -- 172.21.15.81:0/283842252 >> 172.21.15.81:0/283842252 conn(0x7f8b0406ded0 msgr2=0x7f8b040ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 -- 172.21.15.81:0/283842252 shutdown_connections 2024-05-27T06:51:21.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.872+0000 7f8b09800640 1 -- 172.21.15.81:0/283842252 wait complete. 2024-05-27T06:51:21.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 Processor -- start 2024-05-27T06:51:21.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- start start 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b041fa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b04162bd0 0x7f8b04074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0415a3e0 con 0x7f8b040c91c0 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0415a260 con 0x7f8b04162bd0 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b040cf500 con 0x7f8b0406c790 2024-05-27T06:51:21.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48050/0 (socket says 172.21.15.81:48050) 2024-05-27T06:51:21.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 -- 172.21.15.81:0/3118478322 learned_addr learned my addr 172.21.15.81:0/3118478322 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:51:21.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b08e00640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b04162bd0 0x7f8b04074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03e00640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b041fa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 -- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 msgr2=0x7f8b041fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:21.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b041fa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:21.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 -- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b04162bd0 msgr2=0x7f8b04074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b04162bd0 0x7f8b04074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 -- 172.21.15.81:0/3118478322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b04150760 con 0x7f8b040c91c0 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b03400640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8af000d620 tx=0x7f8af000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af000b620 con 0x7f8b040c91c0 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af0019070 con 0x7f8b040c91c0 2024-05-27T06:51:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b04164bb0 con 0x7f8b040c91c0 2024-05-27T06:51:21.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af000b9e0 con 0x7f8b040c91c0 2024-05-27T06:51:21.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.876+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b04074a80 con 0x7f8b040c91c0 2024-05-27T06:51:21.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b04074d90 con 0x7f8b040c91c0 2024-05-27T06:51:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8af000bb80 con 0x7f8b040c91c0 2024-05-27T06:51:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8afbe00640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 0x7f8ae4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:51:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6139+0+0 (secure 0 0 0) 0x7f8af008f6f0 con 0x7f8b040c91c0 2024-05-27T06:51:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8b03e00640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 0x7f8ae4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:51:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.880+0000 7f8b03e00640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 0x7f8ae4071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8af4007a70 tx=0x7f8af40079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:51:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:21.884+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8af005be60 con 0x7f8b040c91c0 2024-05-27T06:51:22.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.076+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b04050870 con 0x7f8b040c91c0 2024-05-27T06:51:22.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8afbe00640 1 -- 172.21.15.81:0/3118478322 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7f8af0060160 con 0x7f8b040c91c0 2024-05-27T06:51:22.074 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:51:22.074 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:51:22.074 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 msgr2=0x7f8ae4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 0x7f8ae4071940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8af4007a70 tx=0x7f8af40079c0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 msgr2=0x7f8b04164670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8af000d620 tx=0x7f8af000dae0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 shutdown_connections 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8ae406f430 0x7f8ae4071940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b04162bd0 0x7f8b04074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b040c91c0 0x7f8b04164670 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 --2- 172.21.15.81:0/3118478322 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b0406c790 0x7f8b041fa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 >> 172.21.15.81:0/3118478322 conn(0x7f8b0406ded0 msgr2=0x7f8b040cdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 shutdown_connections 2024-05-27T06:51:22.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:51:22.080+0000 7f8b09800640 1 -- 172.21.15.81:0/3118478322 wait complete. 2024-05-27T06:51:22.095 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-05-27T06:51:22.096 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 87 secs before thrashing 2024-05-27T06:51:22.129 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:22.132 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:22.393 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:22.408 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:22.409 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:22.409 INFO:tasks.workunit.client.0.smithi081.stdout: 37757 ? 00:00:00 ps 2024-05-27T06:51:22.415 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:22.463 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:22.468 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:22.481 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:22.484 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:22.487 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:22.499 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:22.555 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:22.559 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:22.834 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:22.848 INFO:tasks.workunit.client.0.smithi081.stdout: 37767 ? 00:00:00 ps 2024-05-27T06:51:22.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:22.873 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:22.876 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:22.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:22.885 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:23.190 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:23.222 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:23.840 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:23.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:23.951 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:23.965 INFO:tasks.workunit.client.0.smithi081.stdout: 37777 ? 00:00:00 ps 2024-05-27T06:51:23.969 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:23.990 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:23.994 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:24.000 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:24.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:24.006 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:24.020 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:24.030 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:24.031 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:24.093 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:24.106 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:24.107 INFO:tasks.workunit.client.0.smithi081.stdout: 37787 ? 00:00:00 ps 2024-05-27T06:51:24.111 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:24.131 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:24.135 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:24.141 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:24.144 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:24.146 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:24.158 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:24.169 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:24.171 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:24.250 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:24.265 INFO:tasks.workunit.client.0.smithi081.stdout: 37797 ? 00:00:00 ps 2024-05-27T06:51:24.271 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:24.289 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:24.292 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:24.298 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:24.301 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:24.303 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:24.315 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:24.325 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:24.327 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:24.402 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:24.417 INFO:tasks.workunit.client.0.smithi081.stdout: 37807 ? 00:00:00 ps 2024-05-27T06:51:24.421 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:25.437 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:25.441 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:25.446 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:25.450 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:25.452 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:25.463 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:25.473 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:25.475 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:25.637 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:25.659 INFO:tasks.workunit.client.0.smithi081.stdout: 37818 ? 00:00:00 ps 2024-05-27T06:51:25.664 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:26.188 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:26.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:26.198 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:26.201 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:26.204 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:26.217 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:26.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:26.229 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:26.360 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:26.374 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:26.375 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:26.375 INFO:tasks.workunit.client.0.smithi081.stdout: 37828 ? 00:00:00 ps 2024-05-27T06:51:26.380 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:26.849 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:26.851 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:26.856 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:26.859 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:26.862 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:26.874 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:27.135 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:27.137 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:27.227 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:27.241 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:27.241 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:27.241 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:27.241 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:27.242 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:27.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:27.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:27.242 INFO:tasks.workunit.client.0.smithi081.stdout: 37838 ? 00:00:00 ps 2024-05-27T06:51:27.246 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:27.777 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:27.780 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:27.786 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a 2024-05-27T06:51:27.789 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/one 2024-05-27T06:51:27.791 INFO:tasks.workunit.client.0.smithi081.stderr:+ cat 2024-05-27T06:51:27.803 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir a/.snap/two 2024-05-27T06:51:27.813 INFO:tasks.workunit.client.0.smithi081.stderr:+ chmod 777 a/foo 2024-05-27T06:51:27.815 INFO:tasks.workunit.client.0.smithi081.stderr:+ sync 2024-05-27T06:51:27.961 INFO:tasks.workunit.client.0.smithi081.stderr:+ ps 2024-05-27T06:51:27.975 INFO:tasks.workunit.client.0.smithi081.stdout: PID TTY TIME CMD 2024-05-27T06:51:27.975 INFO:tasks.workunit.client.0.smithi081.stdout: 1211 ? 00:00:00 systemd 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 1224 ? 00:00:00 (sd-pam) 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 17557 ? 00:00:01 sshd 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 33840 ? 00:00:00 bash 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 37570 ? 00:00:00 timeout 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 37573 ? 00:00:00 snaptest-double 2024-05-27T06:51:27.976 INFO:tasks.workunit.client.0.smithi081.stdout: 37848 ? 00:00:00 ps 2024-05-27T06:51:27.982 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a/.snap/one a/.snap/two 2024-05-27T06:51:28.003 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm a/foo 2024-05-27T06:51:28.007 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir a 2024-05-27T06:51:28.012 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:51:28.013 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:28.013 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:28.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:28.087 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-05-27T06:51:28.087 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-05-27T06:51:28.152 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir .snap/foo 2024-05-27T06:51:28.516 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo We want ENOENT, not ESTALE, here. 2024-05-27T06:51:28.517 INFO:tasks.workunit.client.0.smithi081.stdout:We want ENOENT, not ESTALE, here. 2024-05-27T06:51:28.517 INFO:tasks.workunit.client.0.smithi081.stderr:+ seq 1 100 2024-05-27T06:51:28.518 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/1 2024-05-27T06:51:28.518 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.522 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-05-27T06:51:28.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/2 2024-05-27T06:51:28.523 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.527 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-05-27T06:51:28.528 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/3 2024-05-27T06:51:28.528 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.532 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-05-27T06:51:28.533 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/4 2024-05-27T06:51:28.533 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.537 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-05-27T06:51:28.538 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/5 2024-05-27T06:51:28.538 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.543 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-05-27T06:51:28.546 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/6 2024-05-27T06:51:28.547 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.551 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-05-27T06:51:28.552 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/7 2024-05-27T06:51:28.552 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.556 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-05-27T06:51:28.557 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/8 2024-05-27T06:51:28.557 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.561 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-05-27T06:51:28.562 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/9 2024-05-27T06:51:28.562 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.566 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-05-27T06:51:28.566 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/10 2024-05-27T06:51:28.567 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.570 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-05-27T06:51:28.571 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/11 2024-05-27T06:51:28.571 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.575 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-05-27T06:51:28.576 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/12 2024-05-27T06:51:28.576 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:28.579 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-05-27T06:51:28.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/13 2024-05-27T06:51:28.580 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.023 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-05-27T06:51:29.024 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/14 2024-05-27T06:51:29.024 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.029 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-05-27T06:51:29.030 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/15 2024-05-27T06:51:29.030 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.034 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-05-27T06:51:29.035 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/16 2024-05-27T06:51:29.035 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.039 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-05-27T06:51:29.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/17 2024-05-27T06:51:29.040 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.044 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-05-27T06:51:29.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/18 2024-05-27T06:51:29.045 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.049 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-05-27T06:51:29.050 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/19 2024-05-27T06:51:29.050 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.053 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-05-27T06:51:29.054 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/20 2024-05-27T06:51:29.054 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.058 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-05-27T06:51:29.059 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/21 2024-05-27T06:51:29.060 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.063 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-05-27T06:51:29.064 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/22 2024-05-27T06:51:29.064 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.068 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-05-27T06:51:29.069 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/23 2024-05-27T06:51:29.069 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.073 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-05-27T06:51:29.074 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/24 2024-05-27T06:51:29.074 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.078 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-05-27T06:51:29.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/25 2024-05-27T06:51:29.079 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.083 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-05-27T06:51:29.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/26 2024-05-27T06:51:29.084 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.088 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-05-27T06:51:29.089 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/27 2024-05-27T06:51:29.089 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.093 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-05-27T06:51:29.093 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/28 2024-05-27T06:51:29.094 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.097 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-05-27T06:51:29.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/29 2024-05-27T06:51:29.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.102 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-05-27T06:51:29.103 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/30 2024-05-27T06:51:29.103 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.107 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-05-27T06:51:29.107 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/31 2024-05-27T06:51:29.108 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.111 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-05-27T06:51:29.112 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/32 2024-05-27T06:51:29.112 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.116 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-05-27T06:51:29.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/33 2024-05-27T06:51:29.117 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.121 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-05-27T06:51:29.122 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/34 2024-05-27T06:51:29.122 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.127 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-05-27T06:51:29.127 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/35 2024-05-27T06:51:29.127 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.132 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-05-27T06:51:29.133 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/36 2024-05-27T06:51:29.134 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.138 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-05-27T06:51:29.138 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/37 2024-05-27T06:51:29.139 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.142 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-05-27T06:51:29.143 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/38 2024-05-27T06:51:29.143 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.146 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-05-27T06:51:29.147 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/39 2024-05-27T06:51:29.147 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.152 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-05-27T06:51:29.152 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/40 2024-05-27T06:51:29.152 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:29.156 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-05-27T06:51:29.157 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/41 2024-05-27T06:51:29.157 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.150 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-05-27T06:51:30.151 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/42 2024-05-27T06:51:30.151 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.155 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-05-27T06:51:30.156 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/43 2024-05-27T06:51:30.156 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.160 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-05-27T06:51:30.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/44 2024-05-27T06:51:30.161 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.165 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-05-27T06:51:30.166 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/45 2024-05-27T06:51:30.166 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.170 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-05-27T06:51:30.171 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/46 2024-05-27T06:51:30.171 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.175 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-05-27T06:51:30.176 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/47 2024-05-27T06:51:30.176 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.180 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-05-27T06:51:30.181 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/48 2024-05-27T06:51:30.181 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.186 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-05-27T06:51:30.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/49 2024-05-27T06:51:30.187 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.191 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-05-27T06:51:30.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/50 2024-05-27T06:51:30.192 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.195 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-05-27T06:51:30.196 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/51 2024-05-27T06:51:30.196 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.199 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-05-27T06:51:30.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/52 2024-05-27T06:51:30.200 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.204 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-05-27T06:51:30.205 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/53 2024-05-27T06:51:30.205 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.208 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-05-27T06:51:30.209 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/54 2024-05-27T06:51:30.209 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.213 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-05-27T06:51:30.214 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/55 2024-05-27T06:51:30.214 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.219 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-05-27T06:51:30.219 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/56 2024-05-27T06:51:30.220 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.753 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-05-27T06:51:30.753 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/57 2024-05-27T06:51:30.754 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.759 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-05-27T06:51:30.760 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/58 2024-05-27T06:51:30.760 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.764 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-05-27T06:51:30.765 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/59 2024-05-27T06:51:30.765 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.769 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-05-27T06:51:30.770 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/60 2024-05-27T06:51:30.770 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.774 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-05-27T06:51:30.775 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/61 2024-05-27T06:51:30.775 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.779 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-05-27T06:51:30.780 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/62 2024-05-27T06:51:30.780 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.784 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-05-27T06:51:30.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/63 2024-05-27T06:51:30.785 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.789 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-05-27T06:51:30.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/64 2024-05-27T06:51:30.790 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.795 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-05-27T06:51:30.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/65 2024-05-27T06:51:30.796 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.800 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-05-27T06:51:30.801 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/66 2024-05-27T06:51:30.801 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.806 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-05-27T06:51:30.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/67 2024-05-27T06:51:30.807 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.812 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-05-27T06:51:30.813 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/68 2024-05-27T06:51:30.813 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.817 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-05-27T06:51:30.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/69 2024-05-27T06:51:30.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.823 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-05-27T06:51:30.824 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/70 2024-05-27T06:51:30.824 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.830 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-05-27T06:51:30.831 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/71 2024-05-27T06:51:30.831 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.835 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-05-27T06:51:30.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/72 2024-05-27T06:51:30.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.841 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-05-27T06:51:30.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/73 2024-05-27T06:51:30.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.846 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-05-27T06:51:30.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/74 2024-05-27T06:51:30.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.851 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-05-27T06:51:30.853 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/75 2024-05-27T06:51:30.853 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.856 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-05-27T06:51:30.857 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/76 2024-05-27T06:51:30.857 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.861 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-05-27T06:51:30.862 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/77 2024-05-27T06:51:30.862 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.867 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-05-27T06:51:30.868 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/78 2024-05-27T06:51:30.868 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.872 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-05-27T06:51:30.873 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/79 2024-05-27T06:51:30.873 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.877 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-05-27T06:51:30.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/80 2024-05-27T06:51:30.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:30.882 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-05-27T06:51:30.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/81 2024-05-27T06:51:30.883 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.192 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-05-27T06:51:31.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/82 2024-05-27T06:51:31.193 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.811 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-05-27T06:51:31.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/83 2024-05-27T06:51:31.812 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.817 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-05-27T06:51:31.818 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/84 2024-05-27T06:51:31.819 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.823 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-05-27T06:51:31.824 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/85 2024-05-27T06:51:31.824 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.830 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-05-27T06:51:31.830 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/86 2024-05-27T06:51:31.832 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.836 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-05-27T06:51:31.836 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/87 2024-05-27T06:51:31.837 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.841 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-05-27T06:51:31.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/88 2024-05-27T06:51:31.842 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.846 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-05-27T06:51:31.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/89 2024-05-27T06:51:31.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.851 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-05-27T06:51:31.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/90 2024-05-27T06:51:31.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.856 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-05-27T06:51:31.857 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/91 2024-05-27T06:51:31.857 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.862 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-05-27T06:51:31.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/92 2024-05-27T06:51:31.863 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.866 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-05-27T06:51:31.867 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/93 2024-05-27T06:51:31.867 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.871 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-05-27T06:51:31.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/94 2024-05-27T06:51:31.872 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.876 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-05-27T06:51:31.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/95 2024-05-27T06:51:31.878 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.881 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-05-27T06:51:31.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/96 2024-05-27T06:51:31.882 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.886 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-05-27T06:51:31.887 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/97 2024-05-27T06:51:31.887 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.891 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-05-27T06:51:31.892 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/98 2024-05-27T06:51:31.892 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.896 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-05-27T06:51:31.896 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/99 2024-05-27T06:51:31.897 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.901 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-05-27T06:51:31.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ stat .snap/foo/100 2024-05-27T06:51:31.902 INFO:tasks.workunit.client.0.smithi081.stderr:+ grep No such file 2024-05-27T06:51:31.905 INFO:tasks.workunit.client.0.smithi081.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-05-27T06:51:31.906 INFO:tasks.workunit.client.0.smithi081.stderr:+ rmdir .snap/foo 2024-05-27T06:51:31.919 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2024-05-27T06:51:31.920 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2024-05-27T06:51:31.921 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-27T06:51:31.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-27T06:51:31.997 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-05-27T06:51:31.998 DEBUG:teuthology.orchestra.run.smithi081:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-05-27T06:51:32.062 INFO:tasks.workunit.client.0.smithi081.stderr:+ set -e 2024-05-27T06:51:32.062 INFO:tasks.workunit.client.0.smithi081.stderr:+ sudo git config --global http.sslVerify false 2024-05-27T06:51:32.081 INFO:tasks.workunit.client.0.smithi081.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-05-27T06:51:32.100 INFO:tasks.workunit.client.0.smithi081.stderr:+ retried=false 2024-05-27T06:51:32.100 INFO:tasks.workunit.client.0.smithi081.stderr:+ trap -- retry EXIT 2024-05-27T06:51:32.100 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm -rf ceph 2024-05-27T06:51:32.101 INFO:tasks.workunit.client.0.smithi081.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-05-27T06:51:32.108 INFO:tasks.workunit.client.0.smithi081.stderr:Cloning into 'ceph'... 2024-05-27T06:51:33.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:51:33.449 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:52:03.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:52:03.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:52:33.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:52:33.561 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:52:49.097 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:49.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.263+0000 7f5f3a200640 1 Processor -- start 2024-05-27T06:52:49.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.263+0000 7f5f3a200640 1 -- start start 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5f34072430 0x7f5f34159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f3415a040 0x7f5f340c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 0x7f5f3415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f5f340c5040 con 0x7f5f34072430 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5f340c75c0 con 0x7f5f340c60a0 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f3a200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340c7dd0 con 0x7f5f3415a040 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5f34072430 0x7f5f34159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5f34072430 0x7f5f34159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55794/0 (socket says 172.21.15.81:55794) 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 -- 172.21.15.81:0/1178626110 learned_addr learned my addr 172.21.15.81:0/1178626110 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33e00640 1 --2- 172.21.15.81:0/1178626110 >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 0x7f5f3415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f2aa00640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f3415a040 0x7f5f340c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 -- 172.21.15.81:0/1178626110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f3415a040 msgr2=0x7f5f340c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f3415a040 0x7f5f340c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 -- 172.21.15.81:0/1178626110 >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 msgr2=0x7f5f3415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 --2- 172.21.15.81:0/1178626110 >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 0x7f5f3415c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33400640 1 -- 172.21.15.81:0/1178626110 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f34072840 con 0x7f5f34072430 2024-05-27T06:52:49.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.267+0000 7f5f33e00640 1 --2- 172.21.15.81:0/1178626110 >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 0x7f5f3415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:52:49.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f33400640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f34159b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5f3406f9a0 tx=0x7f5f2401e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3ad3d6ef6a06c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f32a00640 1 -- 172.21.15.81:0/1178626110 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240b4070 con 0x7f5f34072430 2024-05-27T06:52:49.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f32a00640 1 -- 172.21.15.81:0/1178626110 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f240a77c0 con 0x7f5f34072430 2024-05-27T06:52:49.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f32a00640 1 -- 172.21.15.81:0/1178626110 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240a7b90 con 0x7f5f34072430 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 -- 172.21.15.81:0/1178626110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 msgr2=0x7f5f34159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f34159b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5f3406f9a0 tx=0x7f5f2401e930 comp rx=0 tx=0).stop 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 -- 172.21.15.81:0/1178626110 shutdown_connections 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 --2- 172.21.15.81:0/1178626110 >> v2:172.21.15.106:3300/0 conn(0x7f5f340c60a0 0x7f5f3415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f3415a040 0x7f5f340c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 --2- 172.21.15.81:0/1178626110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f34159b00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 -- 172.21.15.81:0/1178626110 >> 172.21.15.81:0/1178626110 conn(0x7f5f34070700 msgr2=0x7f5f340cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 -- 172.21.15.81:0/1178626110 shutdown_connections 2024-05-27T06:52:49.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.271+0000 7f5f3a200640 1 -- 172.21.15.81:0/1178626110 wait complete. 2024-05-27T06:52:49.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 Processor -- start 2024-05-27T06:52:49.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- start start 2024-05-27T06:52:49.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 0x7f5f3414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f3415a040 0x7f5f34148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f340ce6f0 con 0x7f5f34072430 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5f341595e0 con 0x7f5f3415a040 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340c5610 con 0x7f5f340c60a0 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55804/0 (socket says 172.21.15.81:55804) 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 -- 172.21.15.81:0/3947984092 learned_addr learned my addr 172.21.15.81:0/3947984092 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33e00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f3415a040 0x7f5f34148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f2aa00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 0x7f5f3414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 -- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 msgr2=0x7f5f3414bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 0x7f5f3414bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 -- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f3415a040 msgr2=0x7f5f34148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f3415a040 0x7f5f34148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 -- 172.21.15.81:0/3947984092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340cd210 con 0x7f5f34072430 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f2aa00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 0x7f5f3414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f33400640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5f2401cfe0 tx=0x7f5f2401b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240b4070 con 0x7f5f34072430 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f240b6070 con 0x7f5f34072430 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f340cd4f0 con 0x7f5f34072430 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f240afe60 con 0x7f5f34072430 2024-05-27T06:52:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f340cbcd0 con 0x7f5f34072430 2024-05-27T06:52:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.275+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3406a560 con 0x7f5f34072430 2024-05-27T06:52:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.279+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5f240a7680 con 0x7f5f34072430 2024-05-27T06:52:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.279+0000 7f5f2be00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 0x7f5f10071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.279+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7f5f241297f0 con 0x7f5f34072430 2024-05-27T06:52:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.279+0000 7f5f2aa00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 0x7f5f10071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.279+0000 7f5f2aa00640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 0x7f5f10071940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5f34159150 tx=0x7f5f1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.283+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5f240187e0 con 0x7f5f34072430 2024-05-27T06:52:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f340ca190 con 0x7f5f34072430 2024-05-27T06:52:49.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f2be00640 1 -- 172.21.15.81:0/3947984092 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7f5f240f6540 con 0x7f5f34072430 2024-05-27T06:52:49.466 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:49.466 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:49.466 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:49.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 msgr2=0x7f5f10071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 0x7f5f10071940 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5f34159150 tx=0x7f5f1c008040 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 msgr2=0x7f5f340cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5f2401cfe0 tx=0x7f5f2401b6d0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 shutdown_connections 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5f340c60a0 0x7f5f3414bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5f1006f430 0x7f5f10071940 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f3415a040 0x7f5f34148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 --2- 172.21.15.81:0/3947984092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f34072430 0x7f5f340cc1f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 >> 172.21.15.81:0/3947984092 conn(0x7f5f34070700 msgr2=0x7f5f340c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 shutdown_connections 2024-05-27T06:52:49.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.475+0000 7f5f3a200640 1 -- 172.21.15.81:0/3947984092 wait complete. 2024-05-27T06:52:49.487 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=2) 2024-05-27T06:52:49.487 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-27T06:52:49.488 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:52:49.505 INFO:tasks.ceph.mds.b:Stopped 2024-05-27T06:52:49.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:49.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.663+0000 7f1edac00640 1 Processor -- start 2024-05-27T06:52:49.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.663+0000 7f1edac00640 1 -- start start 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.663+0000 7f1edac00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1ed4163fb0 0x7f1ed41663d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1edac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed406c910 con 0x7f1ed40ca370 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1edac00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed40ca780 con 0x7f1ed4154980 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1edac00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed40732b0 con 0x7f1ed4163fb0 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1ed40ca370 0x7f1ed415dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53616/0 (socket says 172.21.15.81:53616) 2024-05-27T06:52:49.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3e00640 1 -- 172.21.15.81:0/401317771 learned_addr learned my addr 172.21.15.81:0/401317771 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/401317771 >> v2:172.21.15.106:3300/0 conn(0x7f1ed4163fb0 0x7f1ed41663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 163620428 0 0) 0x7f1ed406c910 con 0x7f1ed40ca370 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb8003530 con 0x7f1ed40ca370 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632522769 0 0) 0x7f1ed40ca780 con 0x7f1ed4154980 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed406c910 con 0x7f1ed4154980 2024-05-27T06:52:49.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2231110472 0 0) 0x7f1eb8003530 con 0x7f1ed40ca370 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 >> v2:172.21.15.106:3300/0 conn(0x7f1ed4163fb0 msgr2=0x7f1ed41663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 --2- 172.21.15.81:0/401317771 >> v2:172.21.15.106:3300/0 conn(0x7f1ed4163fb0 0x7f1ed41663d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 >> v1:172.21.15.106:6789/0 conn(0x7f1ed4154980 legacy=0x7f1ed40c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed406d530 con 0x7f1ed40ca370 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f1ec4001a80 con 0x7f1ed40ca370 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 >> v1:172.21.15.81:6789/0 conn(0x7f1ed40ca370 legacy=0x7f1ed415dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8003c40 0x7f1eb8004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 0x7f1eb8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb8005290 0x7f1eb800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed40732b0 con 0x7f1eb8005290 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb800c240 con 0x7f1eb8004810 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb800c440 con 0x7f1eb8003c40 2024-05-27T06:52:49.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb8005290 0x7f1eb800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 0x7f1eb8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3e00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8003c40 0x7f1eb8004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 -- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8003c40 msgr2=0x7f1eb8004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8003c40 0x7f1eb8004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 -- 172.21.15.81:0/401317771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb8005290 msgr2=0x7f1eb800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb8005290 0x7f1eb800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed3400640 1 -- 172.21.15.81:0/401317771 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb8003530 con 0x7f1eb8004810 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb8005290 0x7f1eb800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:52:49.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.667+0000 7f1ed8c00640 1 -- 172.21.15.81:0/401317771 reap_dead start 2024-05-27T06:52:49.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1ed3400640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 0x7f1eb8004c20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0001900 tx=0x7f1ec001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=847b19644995d657 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0001550 con 0x7f1eb8004810 2024-05-27T06:52:49.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec00052d0 con 0x7f1eb8004810 2024-05-27T06:52:49.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1ed2a00640 1 -- 172.21.15.81:0/401317771 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00054c0 con 0x7f1eb8004810 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 -- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 msgr2=0x7f1eb8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 0x7f1eb8004c20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0001900 tx=0x7f1ec001c460 comp rx=0 tx=0).stop 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 -- 172.21.15.81:0/401317771 shutdown_connections 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 --2- 172.21.15.81:0/401317771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1eb8004810 0x7f1eb8004c20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 -- 172.21.15.81:0/401317771 >> 172.21.15.81:0/401317771 conn(0x7f1ed406f230 msgr2=0x7f1ed4070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 -- 172.21.15.81:0/401317771 shutdown_connections 2024-05-27T06:52:49.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.671+0000 7f1edac00640 1 -- 172.21.15.81:0/401317771 wait complete. 2024-05-27T06:52:49.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 Processor -- start 2024-05-27T06:52:49.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- start start 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8004810 0x7f1ed41f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 0x7f1ed413aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed406d1d0 con 0x7f1ed41464c0 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed4072e20 con 0x7f1ed414f330 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed4068c70 con 0x7f1eb8004810 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55836/0 (socket says 172.21.15.81:55836) 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 -- 172.21.15.81:0/4087611095 learned_addr learned my addr 172.21.15.81:0/4087611095 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed3e00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8004810 0x7f1ed41f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed3400640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 0x7f1ed413aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 -- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8004810 msgr2=0x7f1ed41f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8004810 0x7f1ed41f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 -- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 msgr2=0x7f1ed413aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 0x7f1ed413aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 -- 172.21.15.81:0/4087611095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed41f6cb0 con 0x7f1ed41464c0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed3400640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 0x7f1ed413aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ed8c00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4158340 tx=0x7f1ecc011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0096b0 con 0x7f1ed41464c0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ecc004020 con 0x7f1ed41464c0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc004210 con 0x7f1ed41464c0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed40c6fc0 con 0x7f1ed41464c0 2024-05-27T06:52:49.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.675+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed414ff70 con 0x7f1ed41464c0 2024-05-27T06:52:49.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed406a560 con 0x7f1ed41464c0 2024-05-27T06:52:49.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1ecc01e020 con 0x7f1ed41464c0 2024-05-27T06:52:49.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1ecbe00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 0x7f1eb4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:49.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7f1ecc08eff0 con 0x7f1ed41464c0 2024-05-27T06:52:49.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1ed3e00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 0x7f1eb4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:49.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.679+0000 7f1ed3e00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 0x7f1eb4071910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40015b0 tx=0x7f1ec4001540 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:49.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.683+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1ecc05bd10 con 0x7f1ed41464c0 2024-05-27T06:52:49.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ed406cfb0 con 0x7f1ed41464c0 2024-05-27T06:52:49.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1ecbe00640 1 -- 172.21.15.81:0/4087611095 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7f1ecc05b610 con 0x7f1ed41464c0 2024-05-27T06:52:49.870 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:49.870 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:49.870 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 msgr2=0x7f1eb4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 0x7f1eb4071910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40015b0 tx=0x7f1ec4001540 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 msgr2=0x7f1ed4163180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4158340 tx=0x7f1ecc011e10 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 shutdown_connections 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1eb406f400 0x7f1eb4071910 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ed41464c0 0x7f1ed4163180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ed414f330 0x7f1ed413aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 --2- 172.21.15.81:0/4087611095 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1eb8004810 0x7f1ed41f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 >> 172.21.15.81:0/4087611095 conn(0x7f1ed406f230 msgr2=0x7f1ed4151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 shutdown_connections 2024-05-27T06:52:49.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:49.879+0000 7f1edac00640 1 -- 172.21.15.81:0/4087611095 wait complete. 2024-05-27T06:52:49.891 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:52:51.892 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:52.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.059+0000 7fe999400640 1 Processor -- start 2024-05-27T06:52:52.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.059+0000 7fe999400640 1 -- start start 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe999400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe9940c5b60 0x7fe9940c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe999400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe994072840 con 0x7fe994161cd0 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe999400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9940c5270 con 0x7fe994072430 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe999400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe9940c5040 con 0x7fe9940c5b60 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe993200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe994161cd0 0x7fe9941640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53620/0 (socket says 172.21.15.81:53620) 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe993200640 1 -- 172.21.15.81:0/3781771813 learned_addr learned my addr 172.21.15.81:0/3781771813 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:52:52.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991e00640 1 --2- 172.21.15.81:0/3781771813 >> v2:172.21.15.81:3300/0 conn(0x7fe9940c5b60 0x7fe9940c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:52.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991400640 1 -- 172.21.15.81:0/3781771813 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452250752 0 0) 0x7fe994072840 con 0x7fe994161cd0 2024-05-27T06:52:52.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991400640 1 -- 172.21.15.81:0/3781771813 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe978003530 con 0x7fe994161cd0 2024-05-27T06:52:52.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991e00640 1 -- 172.21.15.81:0/3781771813 >> v1:172.21.15.106:6789/0 conn(0x7fe994072430 legacy=0x7fe994159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991e00640 1 -- 172.21.15.81:0/3781771813 >> v1:172.21.15.81:6789/0 conn(0x7fe994161cd0 legacy=0x7fe9941640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.063+0000 7fe991e00640 1 -- 172.21.15.81:0/3781771813 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9940c7810 con 0x7fe9940c5b60 2024-05-27T06:52:52.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe991e00640 1 --2- 172.21.15.81:0/3781771813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9940c5b60 0x7fe9940c5f70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9800097c0 tx=0x7fe98001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45f7bdafffeab1b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe991400640 1 -- 172.21.15.81:0/3781771813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9800b0070 con 0x7fe9940c5b60 2024-05-27T06:52:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe991400640 1 -- 172.21.15.81:0/3781771813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9800ab600 con 0x7fe9940c5b60 2024-05-27T06:52:52.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe991400640 1 -- 172.21.15.81:0/3781771813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9800ab9d0 con 0x7fe9940c5b60 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 -- 172.21.15.81:0/3781771813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9940c5b60 msgr2=0x7fe9940c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 --2- 172.21.15.81:0/3781771813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9940c5b60 0x7fe9940c5f70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe9800097c0 tx=0x7fe98001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 -- 172.21.15.81:0/3781771813 shutdown_connections 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 --2- 172.21.15.81:0/3781771813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9940c5b60 0x7fe9940c5f70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 -- 172.21.15.81:0/3781771813 >> 172.21.15.81:0/3781771813 conn(0x7fe994070700 msgr2=0x7fe9940cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 -- 172.21.15.81:0/3781771813 shutdown_connections 2024-05-27T06:52:52.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.067+0000 7fe999400640 1 -- 172.21.15.81:0/3781771813 wait complete. 2024-05-27T06:52:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 Processor -- start 2024-05-27T06:52:52.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- start start 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe994072430 0x7fe994158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9940c5b60 0x7fe9941482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9940c7dd0 con 0x7fe994161cd0 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe9940c75c0 con 0x7fe9940c5b60 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe9940c5610 con 0x7fe994072430 2024-05-27T06:52:52.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55846/0 (socket says 172.21.15.81:55846) 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 -- 172.21.15.81:0/2068335958 learned_addr learned my addr 172.21.15.81:0/2068335958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe991e00640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9940c5b60 0x7fe9941482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe992800640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe994072430 0x7fe994158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 -- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe994072430 msgr2=0x7fe994158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe994072430 0x7fe994158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 -- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9940c5b60 msgr2=0x7fe9941482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9940c5b60 0x7fe9941482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 -- 172.21.15.81:0/2068335958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe994151b00 con 0x7fe994161cd0 2024-05-27T06:52:52.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe993200640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00ca70 tx=0x7fe98c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00abd0 con 0x7fe994161cd0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98c00ad70 con 0x7fe994161cd0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c012620 con 0x7fe994161cd0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99415a7f0 con 0x7fe994161cd0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.071+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe994151690 con 0x7fe994161cd0 2024-05-27T06:52:52.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe99406a560 con 0x7fe994161cd0 2024-05-27T06:52:52.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe98c0127c0 con 0x7fe994161cd0 2024-05-27T06:52:52.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe98aa00640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 0x7fe974071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:52.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7fe98c08f9d0 con 0x7fe994161cd0 2024-05-27T06:52:52.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe992800640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 0x7fe974071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:52.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.075+0000 7fe992800640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 0x7fe974071910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe984000c10 tx=0x7fe984009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:52.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.079+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe98c05c720 con 0x7fe994161cd0 2024-05-27T06:52:52.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.267+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9940c5940 con 0x7fe994161cd0 2024-05-27T06:52:52.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe98aa00640 1 -- 172.21.15.81:0/2068335958 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7fe98c05c020 con 0x7fe994161cd0 2024-05-27T06:52:52.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:52.261 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:52.262 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:52.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 msgr2=0x7fe974071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 0x7fe974071910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe984000c10 tx=0x7fe984009040 comp rx=0 tx=0).stop 2024-05-27T06:52:52.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 msgr2=0x7fe994148810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00ca70 tx=0x7fe98c00cf30 comp rx=0 tx=0).stop 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 shutdown_connections 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe97406f400 0x7fe974071910 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe994161cd0 0x7fe994148810 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9940c5b60 0x7fe9941482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 --2- 172.21.15.81:0/2068335958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe994072430 0x7fe994158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 >> 172.21.15.81:0/2068335958 conn(0x7fe994070700 msgr2=0x7fe99406cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 shutdown_connections 2024-05-27T06:52:52.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:52.271+0000 7fe999400640 1 -- 172.21.15.81:0/2068335958 wait complete. 2024-05-27T06:52:52.283 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:52:54.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 Processor -- start 2024-05-27T06:52:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 -- start start 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcb040c9b90 0x7fcb04154870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 0x7fcb04164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb040c9730 con 0x7fcb0415a5c0 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fcb040cba70 con 0x7fcb040c9b90 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb09600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fcb0415e740 con 0x7fcb0415a9d0 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcb0415a5c0 0x7fcb040cebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53632/0 (socket says 172.21.15.81:53632) 2024-05-27T06:52:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03400640 1 -- 172.21.15.81:0/1675362544 learned_addr learned my addr 172.21.15.81:0/1675362544 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:52:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03e00640 1 --2- 172.21.15.81:0/1675362544 >> v2:172.21.15.81:3300/0 conn(0x7fcb040c9b90 0x7fcb04154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb08c00640 1 --2- 172.21.15.81:0/1675362544 >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 0x7fcb04164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb02a00640 1 -- 172.21.15.81:0/1675362544 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177201454 0 0) 0x7fcb040c9730 con 0x7fcb0415a5c0 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb02a00640 1 -- 172.21.15.81:0/1675362544 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcae8003530 con 0x7fcb0415a5c0 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03e00640 1 -- 172.21.15.81:0/1675362544 >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 msgr2=0x7fcb04164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03e00640 1 --2- 172.21.15.81:0/1675362544 >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 0x7fcb04164f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03e00640 1 -- 172.21.15.81:0/1675362544 >> v1:172.21.15.81:6789/0 conn(0x7fcb0415a5c0 legacy=0x7fcb040cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb03e00640 1 -- 172.21.15.81:0/1675362544 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0415ea20 con 0x7fcb040c9b90 2024-05-27T06:52:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.451+0000 7fcb08c00640 1 --2- 172.21.15.81:0/1675362544 >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 0x7fcb04164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb03e00640 1 --2- 172.21.15.81:0/1675362544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb040c9b90 0x7fcb04154870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcb04070780 tx=0x7fcaf401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75fa6c423fcb13fc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb02a00640 1 -- 172.21.15.81:0/1675362544 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40b0070 con 0x7fcb040c9b90 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb02a00640 1 -- 172.21.15.81:0/1675362544 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf40ab600 con 0x7fcb040c9b90 2024-05-27T06:52:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb02a00640 1 -- 172.21.15.81:0/1675362544 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40ab9d0 con 0x7fcb040c9b90 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 -- 172.21.15.81:0/1675362544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb040c9b90 msgr2=0x7fcb04154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 --2- 172.21.15.81:0/1675362544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb040c9b90 0x7fcb04154870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcb04070780 tx=0x7fcaf401c480 comp rx=0 tx=0).stop 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 -- 172.21.15.81:0/1675362544 shutdown_connections 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 --2- 172.21.15.81:0/1675362544 >> v2:172.21.15.106:3300/0 conn(0x7fcb0415a9d0 0x7fcb04164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 --2- 172.21.15.81:0/1675362544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb040c9b90 0x7fcb04154870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 -- 172.21.15.81:0/1675362544 >> 172.21.15.81:0/1675362544 conn(0x7fcb04071390 msgr2=0x7fcb040717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 -- 172.21.15.81:0/1675362544 shutdown_connections 2024-05-27T06:52:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.455+0000 7fcb09600640 1 -- 172.21.15.81:0/1675362544 wait complete. 2024-05-27T06:52:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 Processor -- start 2024-05-27T06:52:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 -- start start 2024-05-27T06:52:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb040c9b90 0x7fcb0415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb0415a9d0 0x7fcb04159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb040751d0 con 0x7fcb0415a5c0 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcb040cf300 con 0x7fcb0415a9d0 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb09600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0414ff40 con 0x7fcb040c9b90 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55854/0 (socket says 172.21.15.81:55854) 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 -- 172.21.15.81:0/2736186285 learned_addr learned my addr 172.21.15.81:0/2736186285 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03e00640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb040c9b90 0x7fcb0415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb08c00640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb0415a9d0 0x7fcb04159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 -- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb040c9b90 msgr2=0x7fcb0415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb040c9b90 0x7fcb0415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 -- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb0415a9d0 msgr2=0x7fcb04159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb0415a9d0 0x7fcb04159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.459+0000 7fcb03400640 1 -- 172.21.15.81:0/2736186285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb04152590 con 0x7fcb0415a5c0 2024-05-27T06:52:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcb03400640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000c450 tx=0x7fcaf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0017070 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf0005cf0 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0414f180 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0007270 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0414f740 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb04050870 con 0x7fcb0415a5c0 2024-05-27T06:52:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcaf001a070 con 0x7fcb0415a5c0 2024-05-27T06:52:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 0x7fcae4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7fcaf008e620 con 0x7fcb0415a5c0 2024-05-27T06:52:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.463+0000 7fcb03e00640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 0x7fcae4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:54.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.467+0000 7fcb03e00640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 0x7fcae4071910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40ad000 tx=0x7fcaf4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:54.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.471+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fcaf005ada0 con 0x7fcb0415a5c0 2024-05-27T06:52:54.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.663+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb04157fe0 con 0x7fcb0415a5c0 2024-05-27T06:52:54.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.663+0000 7fcafbe00640 1 -- 172.21.15.81:0/2736186285 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7fcaf005f0a0 con 0x7fcb0415a5c0 2024-05-27T06:52:54.655 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:54.655 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:54.656 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 msgr2=0x7fcae4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 0x7fcae4071910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40ad000 tx=0x7fcaf4008040 comp rx=0 tx=0).stop 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 msgr2=0x7fcb0414e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000c450 tx=0x7fcaf000c910 comp rx=0 tx=0).stop 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 shutdown_connections 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcae406f400 0x7fcae4071910 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb0415a9d0 0x7fcb04159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb0415a5c0 0x7fcb0414e7c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 --2- 172.21.15.81:0/2736186285 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcb040c9b90 0x7fcb0415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:54.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 >> 172.21.15.81:0/2736186285 conn(0x7fcb04071390 msgr2=0x7fcb040c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:54.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 shutdown_connections 2024-05-27T06:52:54.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:54.667+0000 7fcb09600640 1 -- 172.21.15.81:0/2736186285 wait complete. 2024-05-27T06:52:54.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:52:56.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:56.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 Processor -- start 2024-05-27T06:52:56.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 -- start start 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fcec415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec40755a0 con 0x7fcec40cc930 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec40c7400 con 0x7fcec40c6de0 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcecc600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40ca9f0 con 0x7fcec40cbd40 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fceca600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcec40cc930 0x7fcec4164d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52310/0 (socket says 172.21.15.81:52310) 2024-05-27T06:52:56.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fceca600640 1 -- 172.21.15.81:0/3245548398 learned_addr learned my addr 172.21.15.81:0/3245548398 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:52:56.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec9200640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fcec415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599807821 0 0) 0x7fcec40755a0 con 0x7fcec40cc930 2024-05-27T06:52:56.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceac003530 con 0x7fcec40cc930 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046157477 0 0) 0x7fceac003530 con 0x7fcec40cc930 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 msgr2=0x7fcec415a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fcec415a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 >> v1:172.21.15.106:6789/0 conn(0x7fcec40c6de0 legacy=0x7fcec40ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40c98c0 con 0x7fcec40cc930 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fcebc001a80 con 0x7fcec40cc930 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.847+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 >> v1:172.21.15.81:6789/0 conn(0x7fcec40cc930 legacy=0x7fcec4164d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceac003a90 0x7fceac003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceac004810 0x7fceac004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fceac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40ca9f0 con 0x7fceac003a90 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fceac005580 con 0x7fceac004810 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fceac00bf10 con 0x7fcec40cbd40 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceac003a90 0x7fceac003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9200640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceac004810 0x7fceac004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fceca600640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fceac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 -- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 msgr2=0x7fceac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcec40cbd40 0x7fceac00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 -- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceac004810 msgr2=0x7fceac004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:56.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceac004810 0x7fceac004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 -- 172.21.15.81:0/3245548398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceac003530 con 0x7fceac003a90 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec9c00640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceac003a90 0x7fceac003f00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001790 tx=0x7fceb801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cb74f431aa5f4b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80013e0 con 0x7fceac003a90 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb80052d0 con 0x7fceac003a90 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.851+0000 7fcec3e00640 1 -- 172.21.15.81:0/3245548398 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8003840 con 0x7fceac003a90 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- 172.21.15.81:0/3245548398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceac003a90 msgr2=0x7fceac003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:56.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 --2- 172.21.15.81:0/3245548398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceac003a90 0x7fceac003f00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fceb8001790 tx=0x7fceb801c3f0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fceca600640 1 -- 172.21.15.81:0/3245548398 reap_dead start 2024-05-27T06:52:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- 172.21.15.81:0/3245548398 shutdown_connections 2024-05-27T06:52:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- 172.21.15.81:0/3245548398 >> 172.21.15.81:0/3245548398 conn(0x7fcec4068ab0 msgr2=0x7fcec4068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- 172.21.15.81:0/3245548398 shutdown_connections 2024-05-27T06:52:56.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- 172.21.15.81:0/3245548398 wait complete. 2024-05-27T06:52:56.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 Processor -- start 2024-05-27T06:52:56.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.855+0000 7fcecc600640 1 -- start start 2024-05-27T06:52:56.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceac003a90 0x7fcec4150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec415b5a0 0x7fcec41f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40cb820 con 0x7fcec41f6ae0 2024-05-27T06:52:56.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40cc150 con 0x7fcec415b5a0 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40c95c0 con 0x7fceac003a90 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47168/0 (socket says 172.21.15.81:47168) 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 -- 172.21.15.81:0/4076875133 learned_addr learned my addr 172.21.15.81:0/4076875133 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcec9200640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec415b5a0 0x7fcec41f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcec9c00640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceac003a90 0x7fcec4150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 -- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceac003a90 msgr2=0x7fcec4150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceac003a90 0x7fcec4150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 -- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec415b5a0 msgr2=0x7fcec41f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:56.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec415b5a0 0x7fcec41f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:56.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 -- 172.21.15.81:0/4076875133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec41514c0 con 0x7fcec41f6ae0 2024-05-27T06:52:56.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fceca600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcec4074680 tx=0x7fcebc007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc01c070 con 0x7fcec41f6ae0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcebc011780 con 0x7fcec41f6ae0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec414f9d0 con 0x7fcec41f6ae0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc010040 con 0x7fcec41f6ae0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec4150780 con 0x7fcec41f6ae0 2024-05-27T06:52:56.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.859+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec406a560 con 0x7fcec41f6ae0 2024-05-27T06:52:56.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.863+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcebc01f070 con 0x7fcec41f6ae0 2024-05-27T06:52:56.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.863+0000 7fcec1600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 0x7fcea4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:56.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.863+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7fcebc092220 con 0x7fcec41f6ae0 2024-05-27T06:52:56.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.863+0000 7fcec9c00640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 0x7fcea4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:56.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.863+0000 7fcec9c00640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 0x7fcea4071940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fceb80ad000 tx=0x7fceb80011a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:56.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:56.867+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fcebc05e920 con 0x7fcec41f6ae0 2024-05-27T06:52:57.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.059+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcec40c90c0 con 0x7fcec41f6ae0 2024-05-27T06:52:57.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.059+0000 7fcec1600640 1 -- 172.21.15.81:0/4076875133 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7fcebc062c20 con 0x7fcec41f6ae0 2024-05-27T06:52:57.052 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:57.052 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:57.053 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 msgr2=0x7fcea4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 0x7fcea4071940 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fceb80ad000 tx=0x7fceb80011a0 comp rx=0 tx=0).stop 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 msgr2=0x7fcec414d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcec4074680 tx=0x7fcebc007a80 comp rx=0 tx=0).stop 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 shutdown_connections 2024-05-27T06:52:57.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcea406f430 0x7fcea4071940 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcec41f6ae0 0x7fcec414d7c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcec415b5a0 0x7fcec41f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 --2- 172.21.15.81:0/4076875133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceac003a90 0x7fcec4150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 >> 172.21.15.81:0/4076875133 conn(0x7fcec4068ab0 msgr2=0x7fcec4158050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 shutdown_connections 2024-05-27T06:52:57.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:57.063+0000 7fcecc600640 1 -- 172.21.15.81:0/4076875133 wait complete. 2024-05-27T06:52:57.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:52:59.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:52:59.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 Processor -- start 2024-05-27T06:52:59.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 -- start start 2024-05-27T06:52:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f35c4154970 0x7f35c415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f35c415daf0 0x7f35c40c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c406f0f0 con 0x7f35c40c8410 2024-05-27T06:52:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f35c406eec0 con 0x7f35c415daf0 2024-05-27T06:52:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35ca800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f35c406f3b0 con 0x7f35c4154970 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35baa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f35c415daf0 0x7f35c40c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35baa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f35c415daf0 0x7f35c40c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47182/0 (socket says 172.21.15.81:47182) 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35baa00640 1 -- 172.21.15.81:0/1855582380 learned_addr learned my addr 172.21.15.81:0/1855582380 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.243+0000 7f35c3400640 1 --2- 172.21.15.81:0/1855582380 >> v2:172.21.15.106:3300/0 conn(0x7f35c4154970 0x7f35c415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35baa00640 1 -- 172.21.15.81:0/1855582380 >> v2:172.21.15.106:3300/0 conn(0x7f35c4154970 msgr2=0x7f35c415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35baa00640 1 --2- 172.21.15.81:0/1855582380 >> v2:172.21.15.106:3300/0 conn(0x7f35c4154970 0x7f35c415afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35baa00640 1 -- 172.21.15.81:0/1855582380 >> v1:172.21.15.106:6789/0 conn(0x7f35c40c8410 legacy=0x7f35c406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35baa00640 1 -- 172.21.15.81:0/1855582380 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c40cad30 con 0x7f35c415daf0 2024-05-27T06:52:59.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35baa00640 1 --2- 172.21.15.81:0/1855582380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c415daf0 0x7f35c40c7ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f35c40704c0 tx=0x7f35ac01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4ede4df48fe4ea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:59.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35c2a00640 1 -- 172.21.15.81:0/1855582380 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0b0070 con 0x7f35c415daf0 2024-05-27T06:52:59.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35c2a00640 1 -- 172.21.15.81:0/1855582380 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ac0ab600 con 0x7f35c415daf0 2024-05-27T06:52:59.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.247+0000 7f35c2a00640 1 -- 172.21.15.81:0/1855582380 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0ab9d0 con 0x7f35c415daf0 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- 172.21.15.81:0/1855582380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c415daf0 msgr2=0x7f35c40c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- 172.21.15.81:0/1855582380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c415daf0 0x7f35c40c7ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f35c40704c0 tx=0x7f35ac01c520 comp rx=0 tx=0).stop 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- 172.21.15.81:0/1855582380 shutdown_connections 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- 172.21.15.81:0/1855582380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c415daf0 0x7f35c40c7ed0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- 172.21.15.81:0/1855582380 >> v2:172.21.15.106:3300/0 conn(0x7f35c4154970 0x7f35c415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- 172.21.15.81:0/1855582380 >> 172.21.15.81:0/1855582380 conn(0x7f35c40cb2e0 msgr2=0x7f35c40cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- 172.21.15.81:0/1855582380 shutdown_connections 2024-05-27T06:52:59.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- 172.21.15.81:0/1855582380 wait complete. 2024-05-27T06:52:59.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 Processor -- start 2024-05-27T06:52:59.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- start start 2024-05-27T06:52:59.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 0x7f35c4146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35c415daf0 0x7f35c40cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35c40c9520 con 0x7f35c40c8410 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f35c40c93a0 con 0x7f35c4154970 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.251+0000 7f35ca800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f35c415df00 con 0x7f35c415daf0 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47186/0 (socket says 172.21.15.81:47186) 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 -- 172.21.15.81:0/4132954000 learned_addr learned my addr 172.21.15.81:0/4132954000 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 -- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35c415daf0 msgr2=0x7f35c40cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35c415daf0 0x7f35c40cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 -- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 msgr2=0x7f35c4146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35baa00640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 0x7f35c4146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 0x7f35c4146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 -- 172.21.15.81:0/4132954000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c4074300 con 0x7f35c40c8410 2024-05-27T06:52:59.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35baa00640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 0x7f35c4146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:52:59.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35c3400640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35b400efc0 tx=0x7f35b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:59.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4018020 con 0x7f35c40c8410 2024-05-27T06:52:59.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b4014480 con 0x7f35c40c8410 2024-05-27T06:52:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c4158570 con 0x7f35c40c8410 2024-05-27T06:52:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b4014850 con 0x7f35c40c8410 2024-05-27T06:52:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c4158b30 con 0x7f35c40c8410 2024-05-27T06:52:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c406a560 con 0x7f35c40c8410 2024-05-27T06:52:59.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.255+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f35b401b020 con 0x7f35c40c8410 2024-05-27T06:52:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.259+0000 7f35bbe00640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 0x7f35a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:52:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.259+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7f35b4090240 con 0x7f35c40c8410 2024-05-27T06:52:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.259+0000 7f35baa00640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 0x7f35a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:52:59.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.259+0000 7f35baa00640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 0x7f35a0071910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ad000 tx=0x7f35ac0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:52:59.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.263+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f35b4022030 con 0x7f35c40c8410 2024-05-27T06:52:59.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.451+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35c406fa90 con 0x7f35c40c8410 2024-05-27T06:52:59.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.451+0000 7f35bbe00640 1 -- 172.21.15.81:0/4132954000 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7f35b4010040 con 0x7f35c40c8410 2024-05-27T06:52:59.445 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:52:59.445 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:52:59.445 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:52:59.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 msgr2=0x7f35a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 0x7f35a0071910 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0ad000 tx=0x7f35ac0a6000 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 msgr2=0x7f35c4159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35b400efc0 tx=0x7f35b400c490 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 shutdown_connections 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f35c415daf0 0x7f35c40cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f35a006f400 0x7f35a0071910 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35c4154970 0x7f35c4146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 --2- 172.21.15.81:0/4132954000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35c40c8410 0x7f35c4159330 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 >> 172.21.15.81:0/4132954000 conn(0x7f35c40cb2e0 msgr2=0x7f35c4071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 shutdown_connections 2024-05-27T06:52:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:52:59.455+0000 7f35ca800640 1 -- 172.21.15.81:0/4132954000 wait complete. 2024-05-27T06:52:59.466 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:53:01.466 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:01.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 Processor -- start 2024-05-27T06:53:01.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 -- start start 2024-05-27T06:53:01.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ffa800c5f70 0x7ffa80165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa80072840 con 0x7ffa80072430 2024-05-27T06:53:01.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa800c5270 con 0x7ffa800c5b60 2024-05-27T06:53:01.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.635+0000 7ffa86200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ffa800c5040 con 0x7ffa800c5f70 2024-05-27T06:53:01.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7f400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffa80072430 0x7ffa80159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52324/0 (socket says 172.21.15.81:52324) 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7f400640 1 -- 172.21.15.81:0/1316739458 learned_addr learned my addr 172.21.15.81:0/1316739458 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/1316739458 >> v2:172.21.15.106:3300/0 conn(0x7ffa800c5f70 0x7ffa80165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1950529849 0 0) 0x7ffa80072840 con 0x7ffa80072430 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa64003530 con 0x7ffa80072430 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2550322602 0 0) 0x7ffa64003530 con 0x7ffa80072430 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 >> v2:172.21.15.106:3300/0 conn(0x7ffa800c5f70 msgr2=0x7ffa80165400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 --2- 172.21.15.81:0/1316739458 >> v2:172.21.15.106:3300/0 conn(0x7ffa800c5f70 0x7ffa80165400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 >> v1:172.21.15.106:6789/0 conn(0x7ffa800c5b60 legacy=0x7ffa80161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa800c7810 con 0x7ffa80072430 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7ffa70001a80 con 0x7ffa80072430 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 >> v1:172.21.15.81:6789/0 conn(0x7ffa80072430 legacy=0x7ffa80159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa64003a60 0x7ffa64003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64004810 0x7ffa64004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 0x7ffa6400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6400c1c0 con 0x7ffa64005260 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6400c370 con 0x7ffa64004810 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6400c570 con 0x7ffa64003a60 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 0x7ffa6400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa76a00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64004810 0x7ffa64004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7f400640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa64003a60 0x7ffa64003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 -- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa64003a60 msgr2=0x7ffa64003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa64003a60 0x7ffa64003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 -- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64004810 msgr2=0x7ffa64004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64004810 0x7ffa64004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa7fe00640 1 -- 172.21.15.81:0/1316739458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa64003530 con 0x7ffa64005260 2024-05-27T06:53:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.639+0000 7ffa76a00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64004810 0x7ffa64004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:01.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa7fe00640 1 -- 172.21.15.81:0/1316739458 reap_dead start 2024-05-27T06:53:01.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 0x7ffa6400b9d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009670 tx=0x7ffa7801eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bc6a202ac6b9bf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:01.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa780096b0 con 0x7ffa64005260 2024-05-27T06:53:01.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa78004020 con 0x7ffa64005260 2024-05-27T06:53:01.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa7ea00640 1 -- 172.21.15.81:0/1316739458 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78004410 con 0x7ffa64005260 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 -- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 msgr2=0x7ffa6400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 0x7ffa6400b9d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009670 tx=0x7ffa7801eca0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 -- 172.21.15.81:0/1316739458 shutdown_connections 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 --2- 172.21.15.81:0/1316739458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa64005260 0x7ffa6400b9d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 -- 172.21.15.81:0/1316739458 >> 172.21.15.81:0/1316739458 conn(0x7ffa80070700 msgr2=0x7ffa800cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 -- 172.21.15.81:0/1316739458 shutdown_connections 2024-05-27T06:53:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.643+0000 7ffa86200640 1 -- 172.21.15.81:0/1316739458 wait complete. 2024-05-27T06:53:01.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 Processor -- start 2024-05-27T06:53:01.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- start start 2024-05-27T06:53:01.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64005260 0x7ffa800cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa80149eb0 0x7ffa8014a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800c7dd0 con 0x7ffa801f6220 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ffa800c75c0 con 0x7ffa64005260 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ffa801595e0 con 0x7ffa80149eb0 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47216/0 (socket says 172.21.15.81:47216) 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 -- 172.21.15.81:0/3550645388 learned_addr learned my addr 172.21.15.81:0/3550645388 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa76a00640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa80149eb0 0x7ffa8014a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7f400640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64005260 0x7ffa800cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 -- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa80149eb0 msgr2=0x7ffa8014a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa80149eb0 0x7ffa8014a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 -- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64005260 msgr2=0x7ffa800cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64005260 0x7ffa800cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 -- 172.21.15.81:0/3550645388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8014a800 con 0x7ffa801f6220 2024-05-27T06:53:01.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa7fe00640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009b70 tx=0x7ffa7801ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa780a7ce0 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa780b7070 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78003860 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa800ccac0 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.647+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa800cad60 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.651+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa8006a560 con 0x7ffa801f6220 2024-05-27T06:53:01.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.651+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ffa78003a00 con 0x7ffa801f6220 2024-05-27T06:53:01.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.651+0000 7ffa77e00640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 0x7ffa5c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:01.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.651+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6111+0+0 (secure 0 0 0) 0x7ffa78128460 con 0x7ffa801f6220 2024-05-27T06:53:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.651+0000 7ffa7f400640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 0x7ffa5c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.655+0000 7ffa7f400640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 0x7ffa5c071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001340 tx=0x7ffa70002040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.655+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ffa780c0020 con 0x7ffa801f6220 2024-05-27T06:53:01.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.851+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa801f3730 con 0x7ffa801f6220 2024-05-27T06:53:01.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.851+0000 7ffa77e00640 1 -- 172.21.15.81:0/3550645388 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6904 (secure 0 0 0) 0x7ffa8014a800 con 0x7ffa801f6220 2024-05-27T06:53:01.843 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:01.843 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:51:10.650336+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":4285,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"b","rank":2,"incarnation":12,"state":"up:active","state_seq":8,"addr":"172.21.15.106:6837/13170464","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":13170464},{"type":"v1","addr":"172.21.15.106:6837","nonce":13170464}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4302,4287,4870,4285]},"id":1}]} 2024-05-27T06:53:01.843 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 35 2024-05-27T06:53:01.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 msgr2=0x7ffa5c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 0x7ffa5c071910 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffa70001340 tx=0x7ffa70002040 comp rx=0 tx=0).stop 2024-05-27T06:53:01.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 msgr2=0x7ffa801f6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009b70 tx=0x7ffa7801ba40 comp rx=0 tx=0).stop 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 shutdown_connections 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ffa80149eb0 0x7ffa8014a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ffa5c06f400 0x7ffa5c071910 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa801f6220 0x7ffa801f6630 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 --2- 172.21.15.81:0/3550645388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffa64005260 0x7ffa800cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 >> 172.21.15.81:0/3550645388 conn(0x7ffa80070700 msgr2=0x7ffa8014c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 shutdown_connections 2024-05-27T06:53:01.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:01.855+0000 7ffa86200640 1 -- 172.21.15.81:0/3550645388 wait complete. 2024-05-27T06:53:01.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-05-27T06:53:03.866 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:04.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 Processor -- start 2024-05-27T06:53:04.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 -- start start 2024-05-27T06:53:04.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0b940ca770 0x7f0b940cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b94154200 0x7f0b94154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b940c78b0 con 0x7f0b940cbd00 2024-05-27T06:53:04.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0b94154b50 con 0x7f0b940ca770 2024-05-27T06:53:04.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b99600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0b940c6ee0 con 0x7f0b94154200 2024-05-27T06:53:04.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0b940ca770 0x7f0b940cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0b940ca770 0x7f0b940cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47222/0 (socket says 172.21.15.81:47222) 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 -- 172.21.15.81:0/2166341171 learned_addr learned my addr 172.21.15.81:0/2166341171 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b93400640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b94154200 0x7f0b94154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 -- 172.21.15.81:0/2166341171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b94154200 msgr2=0x7f0b94154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b94154200 0x7f0b94154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 -- 172.21.15.81:0/2166341171 >> v1:172.21.15.106:6789/0 conn(0x7f0b940cbd00 legacy=0x7f0b940cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.035+0000 7f0b92a00640 1 -- 172.21.15.81:0/2166341171 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b940c71c0 con 0x7f0b940ca770 2024-05-27T06:53:04.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b92a00640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b940ca770 0x7f0b940cb7c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009e50 tx=0x7f0b8401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29419ca2801f1c17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b91600640 1 -- 172.21.15.81:0/2166341171 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b840a5810 con 0x7f0b940ca770 2024-05-27T06:53:04.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b91600640 1 -- 172.21.15.81:0/2166341171 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b840a59b0 con 0x7f0b940ca770 2024-05-27T06:53:04.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b91600640 1 -- 172.21.15.81:0/2166341171 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b840a5d80 con 0x7f0b940ca770 2024-05-27T06:53:04.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 -- 172.21.15.81:0/2166341171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b940ca770 msgr2=0x7f0b940cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b940ca770 0x7f0b940cb7c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009e50 tx=0x7f0b8401ca30 comp rx=0 tx=0).stop 2024-05-27T06:53:04.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 -- 172.21.15.81:0/2166341171 shutdown_connections 2024-05-27T06:53:04.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b94154200 0x7f0b94154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 --2- 172.21.15.81:0/2166341171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b940ca770 0x7f0b940cb7c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 -- 172.21.15.81:0/2166341171 >> 172.21.15.81:0/2166341171 conn(0x7f0b94068ae0 msgr2=0x7f0b94068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:04.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 -- 172.21.15.81:0/2166341171 shutdown_connections 2024-05-27T06:53:04.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.039+0000 7f0b99600640 1 -- 172.21.15.81:0/2166341171 wait complete. 2024-05-27T06:53:04.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 Processor -- start 2024-05-27T06:53:04.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 -- start start 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b940ca770 0x7f0b940cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b940cbd00 0x7f0b941511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b940d0640 con 0x7f0b94154200 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0b940d04c0 con 0x7f0b940cbd00 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b99600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0b94167590 con 0x7f0b940ca770 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47238/0 (socket says 172.21.15.81:47238) 2024-05-27T06:53:04.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 -- 172.21.15.81:0/4169913642 learned_addr learned my addr 172.21.15.81:0/4169913642 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b92a00640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b940ca770 0x7f0b940cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b92000640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b940cbd00 0x7f0b941511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 -- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b940ca770 msgr2=0x7f0b940cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b940ca770 0x7f0b940cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 -- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b940cbd00 msgr2=0x7f0b941511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b940cbd00 0x7f0b941511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.043+0000 7f0b93400640 1 -- 172.21.15.81:0/4169913642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9414b4c0 con 0x7f0b94154200 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b93400640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00e7f0 tx=0x7f0b8c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c017070 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c00c940 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9414b1e0 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c00eea0 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b940cdee0 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9406a560 con 0x7f0b94154200 2024-05-27T06:53:04.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0b8c01a070 con 0x7f0b94154200 2024-05-27T06:53:04.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 0x7f0b74071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f0b8c08f420 con 0x7f0b94154200 2024-05-27T06:53:04.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.047+0000 7f0b92a00640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 0x7f0b74071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.051+0000 7f0b92a00640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 0x7f0b74071910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0b840ae000 tx=0x7f0b840a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.055+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0b8c05c0a0 con 0x7f0b94154200 2024-05-27T06:53:04.100 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:53:04.103 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:53:04.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b94157d10 con 0x7f0b94154200 2024-05-27T06:53:04.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b8aa00640 1 -- 172.21.15.81:0/4169913642 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6078 (secure 0 0 0) 0x7f0b8c05b9a0 con 0x7f0b94154200 2024-05-27T06:53:04.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:04.239 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:03.351403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:04.239 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 36 2024-05-27T06:53:04.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 msgr2=0x7f0b74071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 0x7f0b74071910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0b840ae000 tx=0x7f0b840a7070 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 msgr2=0x7f0b940cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00e7f0 tx=0x7f0b8c00ecb0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 shutdown_connections 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0b7406f400 0x7f0b74071910 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b94154200 0x7f0b940cee20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0b940cbd00 0x7f0b941511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 --2- 172.21.15.81:0/4169913642 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0b940ca770 0x7f0b940cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 >> 172.21.15.81:0/4169913642 conn(0x7f0b94068ae0 msgr2=0x7f0b9415b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 shutdown_connections 2024-05-27T06:53:04.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.247+0000 7f0b99600640 1 -- 172.21.15.81:0/4169913642 wait complete. 2024-05-27T06:53:04.260 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-05-27T06:53:04.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:53:04.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:04.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.415+0000 7f864a200640 1 Processor -- start 2024-05-27T06:53:04.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.415+0000 7f864a200640 1 -- start start 2024-05-27T06:53:04.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f864a200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644072430 0x7f8644159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f864a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8644161cd0 0x7f86441640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f864a200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8644072840 con 0x7f864415a040 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f864a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f86440c5040 con 0x7f8644161cd0 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f864a200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f86440c75c0 con 0x7f8644072430 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8644161cd0 0x7f86441640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8644161cd0 0x7f86441640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47242/0 (socket says 172.21.15.81:47242) 2024-05-27T06:53:04.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 -- 172.21.15.81:0/1344896210 learned_addr learned my addr 172.21.15.81:0/1344896210 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643400640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644072430 0x7f8644159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 -- 172.21.15.81:0/1344896210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644072430 msgr2=0x7f8644159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644072430 0x7f8644159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 -- 172.21.15.81:0/1344896210 >> v1:172.21.15.106:6789/0 conn(0x7f864415a040 legacy=0x7f86440c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:04.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.419+0000 7f8643e00640 1 -- 172.21.15.81:0/1344896210 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86440c5270 con 0x7f8644161cd0 2024-05-27T06:53:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f8643e00640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8644161cd0 0x7f86441640f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f863c009e50 tx=0x7f863c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7944ab89346a97 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f8642a00640 1 -- 172.21.15.81:0/1344896210 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c0a5670 con 0x7f8644161cd0 2024-05-27T06:53:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f8642a00640 1 -- 172.21.15.81:0/1344896210 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f863c0a5810 con 0x7f8644161cd0 2024-05-27T06:53:04.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f8642a00640 1 -- 172.21.15.81:0/1344896210 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863c0a5be0 con 0x7f8644161cd0 2024-05-27T06:53:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- 172.21.15.81:0/1344896210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8644161cd0 msgr2=0x7f86441640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8644161cd0 0x7f86441640f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f863c009e50 tx=0x7f863c01ca30 comp rx=0 tx=0).stop 2024-05-27T06:53:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- 172.21.15.81:0/1344896210 shutdown_connections 2024-05-27T06:53:04.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8644161cd0 0x7f86441640f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 --2- 172.21.15.81:0/1344896210 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644072430 0x7f8644159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- 172.21.15.81:0/1344896210 >> 172.21.15.81:0/1344896210 conn(0x7f8644070700 msgr2=0x7f86440cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- 172.21.15.81:0/1344896210 shutdown_connections 2024-05-27T06:53:04.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- 172.21.15.81:0/1344896210 wait complete. 2024-05-27T06:53:04.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 Processor -- start 2024-05-27T06:53:04.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.423+0000 7f864a200640 1 -- start start 2024-05-27T06:53:04.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644072430 0x7f8644147320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644161cd0 0x7f8644158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86440ce6f0 con 0x7f864415a040 2024-05-27T06:53:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f86440c7dd0 con 0x7f8644072430 2024-05-27T06:53:04.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f86440c5610 con 0x7f8644161cd0 2024-05-27T06:53:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f8643400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644072430 0x7f8644147320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47246/0 (socket says 172.21.15.81:47246) 2024-05-27T06:53:04.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 -- 172.21.15.81:0/1164574235 learned_addr learned my addr 172.21.15.81:0/1164574235 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f8643e00640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644161cd0 0x7f8644158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 -- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644161cd0 msgr2=0x7f8644158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644161cd0 0x7f8644158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 -- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644072430 msgr2=0x7f8644147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644072430 0x7f8644147320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 -- 172.21.15.81:0/1164574235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8644146930 con 0x7f864415a040 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863aa00640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f862c00b390 tx=0x7f862c00b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c015020 con 0x7f864415a040 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862c011830 con 0x7f864415a040 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862c011c20 con 0x7f864415a040 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86440cb9d0 con 0x7f864415a040 2024-05-27T06:53:04.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8644145100 con 0x7f864415a040 2024-05-27T06:53:04.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.427+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86441463a0 con 0x7f864415a040 2024-05-27T06:53:04.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.431+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f862c018020 con 0x7f864415a040 2024-05-27T06:53:04.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.431+0000 7f863be00640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 0x7f8620071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:04.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.431+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f862c08f8d0 con 0x7f864415a040 2024-05-27T06:53:04.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.431+0000 7f8643400640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 0x7f8620071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:04.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.431+0000 7f8643400640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 0x7f8620071910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8644070fe0 tx=0x7f8634008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.435+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f862c01f020 con 0x7f864415a040 2024-05-27T06:53:04.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f864406a560 con 0x7f864415a040 2024-05-27T06:53:04.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f863be00640 1 -- 172.21.15.81:0/1164574235 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6078 (secure 0 0 0) 0x7f862c05bf80 con 0x7f864415a040 2024-05-27T06:53:04.618 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:04.618 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:03.351403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:04.619 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 36 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 msgr2=0x7f8620071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 0x7f8620071910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8644070fe0 tx=0x7f8634008040 comp rx=0 tx=0).stop 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 msgr2=0x7f86441588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f862c00b390 tx=0x7f862c00b850 comp rx=0 tx=0).stop 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 shutdown_connections 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8644161cd0 0x7f8644158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f862006f400 0x7f8620071910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f864415a040 0x7f86441588d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 --2- 172.21.15.81:0/1164574235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644072430 0x7f8644147320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:04.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 >> 172.21.15.81:0/1164574235 conn(0x7f8644070700 msgr2=0x7f8644074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:04.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 shutdown_connections 2024-05-27T06:53:04.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:04.627+0000 7f864a200640 1 -- 172.21.15.81:0/1164574235 wait complete. 2024-05-27T06:53:04.641 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:53:03.351403+0000", "qdb_cluster": [ 4287, 4302, 4870, 5049 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:53:06.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:06.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 Processor -- start 2024-05-27T06:53:06.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 -- start start 2024-05-27T06:53:06.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fde8c154970 0x7fde8c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde8c15daf0 0x7fde8c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c06f0f0 con 0x7fde8c0c8410 2024-05-27T06:53:06.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c06eec0 con 0x7fde8c15daf0 2024-05-27T06:53:06.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde90c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c06f3b0 con 0x7fde8c154970 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde81400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde8c15daf0 0x7fde8c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde81400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde8c15daf0 0x7fde8c0c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55464/0 (socket says 172.21.15.81:55464) 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde81400640 1 -- 172.21.15.81:0/1519688080 learned_addr learned my addr 172.21.15.81:0/1519688080 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.811+0000 7fde89e00640 1 --2- 172.21.15.81:0/1519688080 >> v2:172.21.15.106:3300/0 conn(0x7fde8c154970 0x7fde8c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde81400640 1 -- 172.21.15.81:0/1519688080 >> v2:172.21.15.106:3300/0 conn(0x7fde8c154970 msgr2=0x7fde8c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde81400640 1 --2- 172.21.15.81:0/1519688080 >> v2:172.21.15.106:3300/0 conn(0x7fde8c154970 0x7fde8c15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde81400640 1 -- 172.21.15.81:0/1519688080 >> v1:172.21.15.106:6789/0 conn(0x7fde8c0c8410 legacy=0x7fde8c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:06.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde81400640 1 -- 172.21.15.81:0/1519688080 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c0cad30 con 0x7fde8c15daf0 2024-05-27T06:53:06.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde81400640 1 --2- 172.21.15.81:0/1519688080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c15daf0 0x7fde8c0c7ed0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0704c0 tx=0x7fde7401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0437baa7bcdc75f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde89400640 1 -- 172.21.15.81:0/1519688080 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740b0070 con 0x7fde8c15daf0 2024-05-27T06:53:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde89400640 1 -- 172.21.15.81:0/1519688080 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde740ab600 con 0x7fde8c15daf0 2024-05-27T06:53:06.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.815+0000 7fde89400640 1 -- 172.21.15.81:0/1519688080 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740ab9d0 con 0x7fde8c15daf0 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- 172.21.15.81:0/1519688080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c15daf0 msgr2=0x7fde8c0c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- 172.21.15.81:0/1519688080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c15daf0 0x7fde8c0c7ed0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0704c0 tx=0x7fde7401c520 comp rx=0 tx=0).stop 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- 172.21.15.81:0/1519688080 shutdown_connections 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- 172.21.15.81:0/1519688080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c15daf0 0x7fde8c0c7ed0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- 172.21.15.81:0/1519688080 >> v2:172.21.15.106:3300/0 conn(0x7fde8c154970 0x7fde8c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- 172.21.15.81:0/1519688080 >> 172.21.15.81:0/1519688080 conn(0x7fde8c0cb2e0 msgr2=0x7fde8c0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- 172.21.15.81:0/1519688080 shutdown_connections 2024-05-27T06:53:06.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- 172.21.15.81:0/1519688080 wait complete. 2024-05-27T06:53:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 Processor -- start 2024-05-27T06:53:06.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- start start 2024-05-27T06:53:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde8c0c8410 0x7fde8c1f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde8c15daf0 0x7fde8c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.819+0000 7fde90c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0c9520 con 0x7fde8c154970 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde90c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0c93a0 con 0x7fde8c0c8410 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde90c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c15df00 con 0x7fde8c15daf0 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55474/0 (socket says 172.21.15.81:55474) 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 -- 172.21.15.81:0/2634086853 learned_addr learned my addr 172.21.15.81:0/2634086853 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde89e00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde8c0c8410 0x7fde8c1f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde8a800640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde8c15daf0 0x7fde8c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 -- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde8c15daf0 msgr2=0x7fde8c15be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde8c15daf0 0x7fde8c15be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 -- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde8c0c8410 msgr2=0x7fde8c1f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde8c0c8410 0x7fde8c1f6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:06.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 -- 172.21.15.81:0/2634086853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c14fc60 con 0x7fde8c154970 2024-05-27T06:53:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde81400640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fde7401a150 tx=0x7fde740192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740b9070 con 0x7fde8c154970 2024-05-27T06:53:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde740ab7b0 con 0x7fde8c154970 2024-05-27T06:53:06.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c153cd0 con 0x7fde8c154970 2024-05-27T06:53:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740b0040 con 0x7fde8c154970 2024-05-27T06:53:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1f6340 con 0x7fde8c154970 2024-05-27T06:53:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c06a560 con 0x7fde8c154970 2024-05-27T06:53:06.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.823+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fde740bc070 con 0x7fde8c154970 2024-05-27T06:53:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.827+0000 7fde82a00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 0x7fde68071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.827+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7fde74128650 con 0x7fde8c154970 2024-05-27T06:53:06.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.827+0000 7fde89e00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 0x7fde68071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:06.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.827+0000 7fde89e00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 0x7fde68071910 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0c57c0 tx=0x7fde7c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:06.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:06.831+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fde74016110 con 0x7fde8c154970 2024-05-27T06:53:07.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.019+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde8c073c10 con 0x7fde8c154970 2024-05-27T06:53:07.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde82a00640 1 -- 172.21.15.81:0/2634086853 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6078 (secure 0 0 0) 0x7fde740fd350 con 0x7fde8c154970 2024-05-27T06:53:07.013 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:07.013 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:03.351403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:07.013 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 36 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 msgr2=0x7fde68071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 0x7fde68071910 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0c57c0 tx=0x7fde7c009190 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 msgr2=0x7fde8c15d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fde7401a150 tx=0x7fde740192c0 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 shutdown_connections 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fde8c15daf0 0x7fde8c15be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fde6806f400 0x7fde68071910 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c154970 0x7fde8c15d210 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 --2- 172.21.15.81:0/2634086853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fde8c0c8410 0x7fde8c1f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:07.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 >> 172.21.15.81:0/2634086853 conn(0x7fde8c0cb2e0 msgr2=0x7fde8c071700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 shutdown_connections 2024-05-27T06:53:07.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:07.023+0000 7fde90c00640 1 -- 172.21.15.81:0/2634086853 wait complete. 2024-05-27T06:53:07.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:09.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:09.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.218+0000 7f14c3200640 1 Processor -- start 2024-05-27T06:53:09.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.218+0000 7f14c3200640 1 -- start start 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f14c40c9630 0x7f14c4154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c3200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 0x7f14c4164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c3200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14c40c9c70 con 0x7f14c415dac0 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f14c40c9a40 con 0x7f14c40c9630 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c3200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f14c415b0c0 con 0x7f14c415ded0 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f14c40c9630 0x7f14c4154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f14c40c9630 0x7f14c4154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55480/0 (socket says 172.21.15.81:55480) 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 -- 172.21.15.81:0/3450944385 learned_addr learned my addr 172.21.15.81:0/3450944385 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c2800640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 0x7f14c4164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 -- 172.21.15.81:0/3450944385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 msgr2=0x7f14c4164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 0x7f14c4164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 -- 172.21.15.81:0/3450944385 >> v1:172.21.15.106:6789/0 conn(0x7f14c415dac0 legacy=0x7f14c4072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c1e00640 1 -- 172.21.15.81:0/3450944385 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c415a4d0 con 0x7f14c40c9630 2024-05-27T06:53:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.222+0000 7f14c2800640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 0x7f14c4164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:09.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c1e00640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c40c9630 0x7f14c4154780 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009870 tx=0x7f14b401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb91272cbf3a22a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:09.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c0a00640 1 -- 172.21.15.81:0/3450944385 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40aa660 con 0x7f14c40c9630 2024-05-27T06:53:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c0a00640 1 -- 172.21.15.81:0/3450944385 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b40af070 con 0x7f14c40c9630 2024-05-27T06:53:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c0a00640 1 -- 172.21.15.81:0/3450944385 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b40aaa20 con 0x7f14c40c9630 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 -- 172.21.15.81:0/3450944385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c40c9630 msgr2=0x7f14c4154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c40c9630 0x7f14c4154780 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009870 tx=0x7f14b401c500 comp rx=0 tx=0).stop 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 -- 172.21.15.81:0/3450944385 shutdown_connections 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c415ded0 0x7f14c4164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 --2- 172.21.15.81:0/3450944385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c40c9630 0x7f14c4154780 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 -- 172.21.15.81:0/3450944385 >> 172.21.15.81:0/3450944385 conn(0x7f14c4068880 msgr2=0x7f14c4068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 -- 172.21.15.81:0/3450944385 shutdown_connections 2024-05-27T06:53:09.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.226+0000 7f14c3200640 1 -- 172.21.15.81:0/3450944385 wait complete. 2024-05-27T06:53:09.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 Processor -- start 2024-05-27T06:53:09.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 -- start start 2024-05-27T06:53:09.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c40c9630 0x7f14c415bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14c415ded0 0x7f14c414f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f14c415e2e0 con 0x7f14c415dac0 2024-05-27T06:53:09.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f14c40cd280 con 0x7f14c415ded0 2024-05-27T06:53:09.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c3200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f14c4072c30 con 0x7f14c40c9630 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55486/0 (socket says 172.21.15.81:55486) 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 -- 172.21.15.81:0/701349447 learned_addr learned my addr 172.21.15.81:0/701349447 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1e00640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c40c9630 0x7f14c415bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c2800640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14c415ded0 0x7f14c414f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 -- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c40c9630 msgr2=0x7f14c415bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c40c9630 0x7f14c415bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 -- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14c415ded0 msgr2=0x7f14c414f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14c415ded0 0x7f14c414f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 -- 172.21.15.81:0/701349447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c415c060 con 0x7f14c415dac0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14c1400640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c370 tx=0x7f14b000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.230+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0017070 con 0x7f14c415dac0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b00052f0 con 0x7f14c415dac0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0003840 con 0x7f14c415dac0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c4155e90 con 0x7f14c415dac0 2024-05-27T06:53:09.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c41f6580 con 0x7f14c415dac0 2024-05-27T06:53:09.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c4050870 con 0x7f14c415dac0 2024-05-27T06:53:09.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f14b001a070 con 0x7f14c415dac0 2024-05-27T06:53:09.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.234+0000 7f14ba000640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 0x7f14a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:09.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.238+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f14b008e7c0 con 0x7f14c415dac0 2024-05-27T06:53:09.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.238+0000 7f14c1e00640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 0x7f14a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:09.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.238+0000 7f14c1e00640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 0x7f14a4071910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f14b40ae000 tx=0x7f14b40a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:09.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.242+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f14b005adf0 con 0x7f14c415dac0 2024-05-27T06:53:09.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.486+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14c41f6030 con 0x7f14c415dac0 2024-05-27T06:53:09.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.486+0000 7f14ba000640 1 -- 172.21.15.81:0/701349447 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6080 (secure 0 0 0) 0x7f14b005f0f0 con 0x7f14c415dac0 2024-05-27T06:53:09.480 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:09.480 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:08.366578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:resolve","state_seq":29,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,4287]},"id":1}]} 2024-05-27T06:53:09.480 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 37 2024-05-27T06:53:09.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 msgr2=0x7f14a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 0x7f14a4071910 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f14b40ae000 tx=0x7f14b40a7000 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 msgr2=0x7f14c41549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f14b000c370 tx=0x7f14b000c830 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 shutdown_connections 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f14a406f400 0x7f14a4071910 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14c415ded0 0x7f14c414f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c415dac0 0x7f14c41549f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 --2- 172.21.15.81:0/701349447 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14c40c9630 0x7f14c415bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 >> 172.21.15.81:0/701349447 conn(0x7f14c4068880 msgr2=0x7f14c4150470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 shutdown_connections 2024-05-27T06:53:09.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:09.490+0000 7f14c3200640 1 -- 172.21.15.81:0/701349447 wait complete. 2024-05-27T06:53:09.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:11.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 Processor -- start 2024-05-27T06:53:11.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 -- start start 2024-05-27T06:53:11.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f17940c6a90 0x7f17940c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f17940c6fe0 0x7f17940c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17940c73f0 con 0x7f1794075110 2024-05-27T06:53:11.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1794072780 con 0x7f17940c6fe0 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179b000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17940c5b50 con 0x7f17940c6a90 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f17940c6fe0 0x7f17940c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f17940c6fe0 0x7f17940c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55494/0 (socket says 172.21.15.81:55494) 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 -- 172.21.15.81:0/2345323013 learned_addr learned my addr 172.21.15.81:0/2345323013 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f1799c00640 1 --2- 172.21.15.81:0/2345323013 >> v2:172.21.15.106:3300/0 conn(0x7f17940c6a90 0x7f17940c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 -- 172.21.15.81:0/2345323013 >> v2:172.21.15.106:3300/0 conn(0x7f17940c6a90 msgr2=0x7f17940c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 --2- 172.21.15.81:0/2345323013 >> v2:172.21.15.106:3300/0 conn(0x7f17940c6a90 0x7f17940c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 -- 172.21.15.81:0/2345323013 >> v1:172.21.15.106:6789/0 conn(0x7f1794075110 legacy=0x7f1794153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.686+0000 7f179a600640 1 -- 172.21.15.81:0/2345323013 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1794073e40 con 0x7f17940c6fe0 2024-05-27T06:53:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.690+0000 7f179a600640 1 --2- 172.21.15.81:0/2345323013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17940c6fe0 0x7f17940c9180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1794070db0 tx=0x7f178c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93276731b3348b4b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.690+0000 7f1793e00640 1 -- 172.21.15.81:0/2345323013 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c0a5730 con 0x7f17940c6fe0 2024-05-27T06:53:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.690+0000 7f1793e00640 1 -- 172.21.15.81:0/2345323013 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178c0a58d0 con 0x7f17940c6fe0 2024-05-27T06:53:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.690+0000 7f1793e00640 1 -- 172.21.15.81:0/2345323013 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178c0a5ca0 con 0x7f17940c6fe0 2024-05-27T06:53:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- 172.21.15.81:0/2345323013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17940c6fe0 msgr2=0x7f17940c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 --2- 172.21.15.81:0/2345323013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17940c6fe0 0x7f17940c9180 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1794070db0 tx=0x7f178c01c960 comp rx=0 tx=0).stop 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- 172.21.15.81:0/2345323013 shutdown_connections 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 --2- 172.21.15.81:0/2345323013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17940c6fe0 0x7f17940c9180 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 --2- 172.21.15.81:0/2345323013 >> v2:172.21.15.106:3300/0 conn(0x7f17940c6a90 0x7f17940c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- 172.21.15.81:0/2345323013 >> 172.21.15.81:0/2345323013 conn(0x7f179406d180 msgr2=0x7f17940cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- 172.21.15.81:0/2345323013 shutdown_connections 2024-05-27T06:53:11.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- 172.21.15.81:0/2345323013 wait complete. 2024-05-27T06:53:11.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 Processor -- start 2024-05-27T06:53:11.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.694+0000 7f179b000640 1 -- start start 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17940c6a90 0x7f1794152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17940c6fe0 0x7f1794147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17941666b0 con 0x7f1794075110 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1794075520 con 0x7f17940c6a90 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1794166830 con 0x7f17940c6fe0 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55506/0 (socket says 172.21.15.81:55506) 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 -- 172.21.15.81:0/2814957208 learned_addr learned my addr 172.21.15.81:0/2814957208 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799200640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17940c6a90 0x7f1794152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179a600640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17940c6fe0 0x7f1794147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 -- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17940c6fe0 msgr2=0x7f1794147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17940c6fe0 0x7f1794147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 -- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17940c6a90 msgr2=0x7f1794152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17940c6a90 0x7f1794152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 -- 172.21.15.81:0/2814957208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17941f7bb0 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1799c00640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f178800e870 tx=0x7f178800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788017070 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178800c940 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1794157650 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178800ccf0 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1794157c10 con 0x7f1794075110 2024-05-27T06:53:11.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.698+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1794050870 con 0x7f1794075110 2024-05-27T06:53:11.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.702+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f178801a070 con 0x7f1794075110 2024-05-27T06:53:11.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.702+0000 7f1791600640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 0x7f1774071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.702+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f178808f640 con 0x7f1794075110 2024-05-27T06:53:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.702+0000 7f1799200640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 0x7f1774071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:11.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.702+0000 7f1799200640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 0x7f1774071940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1794156e30 tx=0x7f1780009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:11.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.706+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f178805bd00 con 0x7f1794075110 2024-05-27T06:53:11.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.970+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17940c7b20 con 0x7f1794075110 2024-05-27T06:53:11.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.970+0000 7f1791600640 1 -- 172.21.15.81:0/2814957208 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6080 (secure 0 0 0) 0x7f1788060000 con 0x7f1794075110 2024-05-27T06:53:11.963 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:11.963 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:08.366578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:resolve","state_seq":29,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,4287]},"id":1}]} 2024-05-27T06:53:11.963 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 37 2024-05-27T06:53:11.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 msgr2=0x7f1774071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 0x7f1774071940 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1794156e30 tx=0x7f1780009040 comp rx=0 tx=0).stop 2024-05-27T06:53:11.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 msgr2=0x7f17941f7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:11.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f178800e870 tx=0x7f178800ec70 comp rx=0 tx=0).stop 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 shutdown_connections 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17940c6fe0 0x7f1794147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f177406f430 0x7f1774071940 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17940c6a90 0x7f1794152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 --2- 172.21.15.81:0/2814957208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1794075110 0x7f17941f7670 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 >> 172.21.15.81:0/2814957208 conn(0x7f179406d180 msgr2=0x7f1794070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 shutdown_connections 2024-05-27T06:53:11.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:11.974+0000 7f179b000640 1 -- 172.21.15.81:0/2814957208 wait complete. 2024-05-27T06:53:11.987 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:13.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:14.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.166+0000 7f7ce7200640 1 Processor -- start 2024-05-27T06:53:14.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.166+0000 7f7ce7200640 1 -- start start 2024-05-27T06:53:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce7200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7ce00caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce006e3e0 con 0x7f7ce015d300 2024-05-27T06:53:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce7200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce006e6c0 con 0x7f7ce00cfdc0 2024-05-27T06:53:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce7200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce015ce70 con 0x7f7ce00ca2f0 2024-05-27T06:53:14.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdf400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7ce015d300 0x7f7ce0160710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33784/0 (socket says 172.21.15.81:33784) 2024-05-27T06:53:14.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdf400640 1 -- 172.21.15.81:0/2360116172 learned_addr learned my addr 172.21.15.81:0/2360116172 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:14.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7ce00caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 473355874 0 0) 0x7f7ce006e3e0 con 0x7f7ce015d300 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc4003530 con 0x7f7ce015d300 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832890520 0 0) 0x7f7ce006e6c0 con 0x7f7ce00cfdc0 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ce006e3e0 con 0x7f7ce00cfdc0 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2549603604 0 0) 0x7f7cc4003530 con 0x7f7ce015d300 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 msgr2=0x7f7ce00caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7ce00caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 >> v1:172.21.15.106:6789/0 conn(0x7f7ce00cfdc0 legacy=0x7f7ce0167090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce006e9a0 con 0x7f7ce015d300 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f7ccc001680 con 0x7f7ce015d300 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 >> v1:172.21.15.81:6789/0 conn(0x7f7ce015d300 legacy=0x7f7ce0160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7cc4003c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc4004220 0x7f7cc4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7cc4005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce015ce70 con 0x7f7cc4004dc0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc400bd20 con 0x7f7cc4004220 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc400bf20 con 0x7f7ce00ca2f0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7cc4003c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdf400640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc4004220 0x7f7cc4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7cc4005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 -- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 msgr2=0x7f7cc4003c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7cc4003c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 -- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc4004220 msgr2=0x7f7cc4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7cc4004220 0x7f7cc4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7ce5200640 1 -- 172.21.15.81:0/2360116172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4003530 con 0x7f7cc4004dc0 2024-05-27T06:53:14.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.170+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00ca2f0 0x7f7cc4003c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:53:14.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce5200640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7cc4005290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80020b0 tx=0x7f7cd801f090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576c3b9427c796b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:14.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8001c50 con 0x7f7cc4004dc0 2024-05-27T06:53:14.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd8006d60 con 0x7f7cc4004dc0 2024-05-27T06:53:14.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7cdea00640 1 -- 172.21.15.81:0/2360116172 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd80a8d00 con 0x7f7cc4004dc0 2024-05-27T06:53:14.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 -- 172.21.15.81:0/2360116172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 msgr2=0x7f7cc4005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 --2- 172.21.15.81:0/2360116172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7cc4005290 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80020b0 tx=0x7f7cd801f090 comp rx=0 tx=0).stop 2024-05-27T06:53:14.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce5200640 1 -- 172.21.15.81:0/2360116172 reap_dead start 2024-05-27T06:53:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 -- 172.21.15.81:0/2360116172 shutdown_connections 2024-05-27T06:53:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 -- 172.21.15.81:0/2360116172 >> 172.21.15.81:0/2360116172 conn(0x7f7ce0068ae0 msgr2=0x7f7ce0068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 -- 172.21.15.81:0/2360116172 shutdown_connections 2024-05-27T06:53:14.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.174+0000 7f7ce7200640 1 -- 172.21.15.81:0/2360116172 wait complete. 2024-05-27T06:53:14.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.178+0000 7f7ce7200640 1 Processor -- start 2024-05-27T06:53:14.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.178+0000 7f7ce7200640 1 -- start start 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.178+0000 7f7ce7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.178+0000 7f7ce7200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00c8070 0x7f7ce015e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 0x7f7ce00c8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce015cb00 con 0x7f7cc4004dc0 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce015d710 con 0x7f7ce014c0f0 2024-05-27T06:53:14.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce006f700 con 0x7f7ce00c8070 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55524/0 (socket says 172.21.15.81:55524) 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 -- 172.21.15.81:0/3090772062 learned_addr learned my addr 172.21.15.81:0/3090772062 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdf400640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00c8070 0x7f7ce015e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce5200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 0x7f7ce00c8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 -- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00c8070 msgr2=0x7f7ce015e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00c8070 0x7f7ce015e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 -- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 msgr2=0x7f7ce00c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 0x7f7ce00c8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 -- 172.21.15.81:0/3090772062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce014d720 con 0x7f7cc4004dc0 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce5200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 0x7f7ce00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:53:14.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cdfe00640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0011ca0 tx=0x7f7cd000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:14.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd000fcc0 con 0x7f7cc4004dc0 2024-05-27T06:53:14.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd000fe60 con 0x7f7cc4004dc0 2024-05-27T06:53:14.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce015dd50 con 0x7f7cc4004dc0 2024-05-27T06:53:14.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0004260 con 0x7f7cc4004dc0 2024-05-27T06:53:14.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce014cc40 con 0x7f7cc4004dc0 2024-05-27T06:53:14.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.182+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce006a560 con 0x7f7cc4004dc0 2024-05-27T06:53:14.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.186+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7cd0012040 con 0x7f7cc4004dc0 2024-05-27T06:53:14.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.186+0000 7f7cd7e00640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 0x7f7cc0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:14.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.186+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f7cd008fa20 con 0x7f7cc4004dc0 2024-05-27T06:53:14.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.186+0000 7f7cdf400640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 0x7f7cc0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:14.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.186+0000 7f7cdf400640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 0x7f7cc0071940 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ce01f4150 tx=0x7f7ccc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:14.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.190+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7cd0019030 con 0x7f7cc4004dc0 2024-05-27T06:53:14.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.438+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ce00c9d10 con 0x7f7cc4004dc0 2024-05-27T06:53:14.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.438+0000 7f7cd7e00640 1 -- 172.21.15.81:0/3090772062 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6080 (secure 0 0 0) 0x7f7cd005c730 con 0x7f7cc4004dc0 2024-05-27T06:53:14.432 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:14.432 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:08.366578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:resolve","state_seq":29,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,4287]},"id":1}]} 2024-05-27T06:53:14.433 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 37 2024-05-27T06:53:14.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 msgr2=0x7f7cc0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 0x7f7cc0071940 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ce01f4150 tx=0x7f7ccc009040 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 msgr2=0x7f7ce01f6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0011ca0 tx=0x7f7cd000f610 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 shutdown_connections 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ce00c8070 0x7f7ce015e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7cc006f430 0x7f7cc0071940 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce014c0f0 0x7f7ce00c8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 --2- 172.21.15.81:0/3090772062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cc4004dc0 0x7f7ce01f6590 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 >> 172.21.15.81:0/3090772062 conn(0x7f7ce0068ae0 msgr2=0x7f7ce00cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:14.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 shutdown_connections 2024-05-27T06:53:14.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:14.442+0000 7f7ce7200640 1 -- 172.21.15.81:0/3090772062 wait complete. 2024-05-27T06:53:14.456 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:16.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:16.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.634+0000 7f117f200640 1 Processor -- start 2024-05-27T06:53:16.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.634+0000 7f117f200640 1 -- start start 2024-05-27T06:53:16.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117f200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 0x7f11800cb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f1180167000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117f200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11800ca240 con 0x7f11800cb8a0 2024-05-27T06:53:16.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117f200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f11800cbcb0 con 0x7f11800cc7b0 2024-05-27T06:53:16.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f118015d520 con 0x7f1180074c40 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117e800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f1180167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 0x7f11800cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117d400640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f11800cb8a0 0x7f1180074530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:35974/0 (socket says 172.21.15.81:35974) 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117d400640 1 -- 172.21.15.81:0/3573205731 learned_addr learned my addr 172.21.15.81:0/3573205731 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117e800640 1 -- 172.21.15.81:0/3573205731 >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 msgr2=0x7f11800cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:16.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117e800640 1 --2- 172.21.15.81:0/3573205731 >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 0x7f11800cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117e800640 1 -- 172.21.15.81:0/3573205731 >> v1:172.21.15.106:6789/0 conn(0x7f11800cb8a0 legacy=0x7f1180074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117e800640 1 -- 172.21.15.81:0/3573205731 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11800ca4e0 con 0x7f1180074c40 2024-05-27T06:53:16.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.638+0000 7f117de00640 1 --2- 172.21.15.81:0/3573205731 >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 0x7f11800cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:53:16.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117e800640 1 --2- 172.21.15.81:0/3573205731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f1180167000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11800cbe30 tx=0x7f117801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1432a552c13d995d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117ca00640 1 -- 172.21.15.81:0/3573205731 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11780b4070 con 0x7f1180074c40 2024-05-27T06:53:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117ca00640 1 -- 172.21.15.81:0/3573205731 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11780af600 con 0x7f1180074c40 2024-05-27T06:53:16.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117ca00640 1 -- 172.21.15.81:0/3573205731 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11780af9d0 con 0x7f1180074c40 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 -- 172.21.15.81:0/3573205731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 msgr2=0x7f1180167000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 --2- 172.21.15.81:0/3573205731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f1180167000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11800cbe30 tx=0x7f117801f130 comp rx=0 tx=0).stop 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 -- 172.21.15.81:0/3573205731 shutdown_connections 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 --2- 172.21.15.81:0/3573205731 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f1180167000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 --2- 172.21.15.81:0/3573205731 >> v2:172.21.15.106:3300/0 conn(0x7f11800cc7b0 0x7f11800cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 -- 172.21.15.81:0/3573205731 >> 172.21.15.81:0/3573205731 conn(0x7f11800686b0 msgr2=0x7f1180068ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 -- 172.21.15.81:0/3573205731 shutdown_connections 2024-05-27T06:53:16.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.642+0000 7f117f200640 1 -- 172.21.15.81:0/3573205731 wait complete. 2024-05-27T06:53:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 Processor -- start 2024-05-27T06:53:16.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 -- start start 2024-05-27T06:53:16.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f11801e7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800cc7b0 0x7f118014db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f11800ca7c0 con 0x7f11800cb8a0 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f118006e610 con 0x7f11800cc7b0 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.646+0000 7f117f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f11800cb0a0 con 0x7f1180074c40 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50660/0 (socket says 172.21.15.81:50660) 2024-05-27T06:53:16.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 -- 172.21.15.81:0/1747186790 learned_addr learned my addr 172.21.15.81:0/1747186790 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:16.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117e800640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800cc7b0 0x7f118014db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117de00640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f11801e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 -- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 msgr2=0x7f11801e7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f11801e7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 -- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800cc7b0 msgr2=0x7f118014db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800cc7b0 0x7f118014db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 -- 172.21.15.81:0/1747186790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11801e7c40 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117d400640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c3c0 tx=0x7f116c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c017070 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116c005cf0 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11801e8840 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c007310 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118014f410 con 0x7f11800cb8a0 2024-05-27T06:53:16.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.650+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11801e85b0 con 0x7f11800cb8a0 2024-05-27T06:53:16.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.654+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f116c01a070 con 0x7f11800cb8a0 2024-05-27T06:53:16.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.654+0000 7f1176000640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 0x7f1160071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:16.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.654+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f116c08e8b0 con 0x7f11800cb8a0 2024-05-27T06:53:16.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.654+0000 7f117de00640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 0x7f1160071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:16.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.654+0000 7f117de00640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 0x7f1160071910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1170009a50 tx=0x7f1170008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:16.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.658+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f116c05b530 con 0x7f11800cb8a0 2024-05-27T06:53:16.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1180050870 con 0x7f11800cb8a0 2024-05-27T06:53:16.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f1176000640 1 -- 172.21.15.81:0/1747186790 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6082 (secure 0 0 0) 0x7f116c05ae30 con 0x7f11800cb8a0 2024-05-27T06:53:16.918 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:16.918 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:14.089849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:16.918 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 38 2024-05-27T06:53:16.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 msgr2=0x7f1160071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 0x7f1160071910 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1170009a50 tx=0x7f1170008040 comp rx=0 tx=0).stop 2024-05-27T06:53:16.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 msgr2=0x7f1180151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:16.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c3c0 tx=0x7f116c00c880 comp rx=0 tx=0).stop 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 shutdown_connections 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f400 0x7f1160071910 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800cc7b0 0x7f118014db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800cb8a0 0x7f1180151040 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 --2- 172.21.15.81:0/1747186790 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1180074c40 0x7f11801e7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 >> 172.21.15.81:0/1747186790 conn(0x7f11800686b0 msgr2=0x7f1180156660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 shutdown_connections 2024-05-27T06:53:16.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:16.926+0000 7f117f200640 1 -- 172.21.15.81:0/1747186790 wait complete. 2024-05-27T06:53:16.941 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4287": { "addr": "172.21.15.81:6839/328699254", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 328699254, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 328699254, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4287, "incarnation": 10, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:53:14.089849+0000", "qdb_cluster": [ 4287, 4302, 4870, 5049 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4287, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:53:18.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:19.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 Processor -- start 2024-05-27T06:53:19.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 -- start start 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc98815cb80 0x7fc988152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc9880c8f90 0x7fc98806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9880c9f80 con 0x7fc98806c790 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fc9880c58b0 con 0x7fc9880c8f90 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc98ce00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc9880c83f0 con 0x7fc98815cb80 2024-05-27T06:53:19.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc986800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc9880c8f90 0x7fc98806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc986800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc9880c8f90 0x7fc98806cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50664/0 (socket says 172.21.15.81:50664) 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc986800640 1 -- 172.21.15.81:0/1980841987 learned_addr learned my addr 172.21.15.81:0/1980841987 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.110+0000 7fc985e00640 1 --2- 172.21.15.81:0/1980841987 >> v2:172.21.15.106:3300/0 conn(0x7fc98815cb80 0x7fc988152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc986800640 1 -- 172.21.15.81:0/1980841987 >> v2:172.21.15.106:3300/0 conn(0x7fc98815cb80 msgr2=0x7fc988152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc986800640 1 --2- 172.21.15.81:0/1980841987 >> v2:172.21.15.106:3300/0 conn(0x7fc98815cb80 0x7fc988152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc986800640 1 -- 172.21.15.81:0/1980841987 >> v1:172.21.15.106:6789/0 conn(0x7fc98806c790 legacy=0x7fc9880c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc986800640 1 -- 172.21.15.81:0/1980841987 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9880c52f0 con 0x7fc9880c8f90 2024-05-27T06:53:19.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc986800640 1 --2- 172.21.15.81:0/1980841987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9880c8f90 0x7fc98806cba0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc98806d3d0 tx=0x7fc98001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eb777266301d7d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:19.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc984a00640 1 -- 172.21.15.81:0/1980841987 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800a5530 con 0x7fc9880c8f90 2024-05-27T06:53:19.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc984a00640 1 -- 172.21.15.81:0/1980841987 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9800a56d0 con 0x7fc9880c8f90 2024-05-27T06:53:19.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.114+0000 7fc984a00640 1 -- 172.21.15.81:0/1980841987 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800a5aa0 con 0x7fc9880c8f90 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- 172.21.15.81:0/1980841987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9880c8f90 msgr2=0x7fc98806cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 --2- 172.21.15.81:0/1980841987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9880c8f90 0x7fc98806cba0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc98806d3d0 tx=0x7fc98001c9f0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- 172.21.15.81:0/1980841987 shutdown_connections 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 --2- 172.21.15.81:0/1980841987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9880c8f90 0x7fc98806cba0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 --2- 172.21.15.81:0/1980841987 >> v2:172.21.15.106:3300/0 conn(0x7fc98815cb80 0x7fc988152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- 172.21.15.81:0/1980841987 >> 172.21.15.81:0/1980841987 conn(0x7fc988070700 msgr2=0x7fc9880cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- 172.21.15.81:0/1980841987 shutdown_connections 2024-05-27T06:53:19.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- 172.21.15.81:0/1980841987 wait complete. 2024-05-27T06:53:19.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 Processor -- start 2024-05-27T06:53:19.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.118+0000 7fc98ce00640 1 -- start start 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9880c8f90 0x7fc9881491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98815cb80 0x7fc98815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc9880c9c80 con 0x7fc98806c790 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc9880c9b00 con 0x7fc98815cb80 2024-05-27T06:53:19.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc988167810 con 0x7fc9880c8f90 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50674/0 (socket says 172.21.15.81:50674) 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 -- 172.21.15.81:0/2633051844 learned_addr learned my addr 172.21.15.81:0/2633051844 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc986800640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98815cb80 0x7fc98815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985400640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9880c8f90 0x7fc9881491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 -- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9880c8f90 msgr2=0x7fc9881491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9880c8f90 0x7fc9881491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 -- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98815cb80 msgr2=0x7fc98815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98815cb80 0x7fc98815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 -- 172.21.15.81:0/2633051844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc988149720 con 0x7fc98806c790 2024-05-27T06:53:19.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc985e00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc97800d5f0 tx=0x7fc97800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9780125f0 con 0x7fc98806c790 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc978019070 con 0x7fc98806c790 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc988159ce0 con 0x7fc98806c790 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9780129b0 con 0x7fc98806c790 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc988148420 con 0x7fc98806c790 2024-05-27T06:53:19.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.122+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc98806a560 con 0x7fc98806c790 2024-05-27T06:53:19.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.126+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc978012b50 con 0x7fc98806c790 2024-05-27T06:53:19.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.126+0000 7fc97e000640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 0x7fc968071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:19.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.126+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7fc97808f720 con 0x7fc98806c790 2024-05-27T06:53:19.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.126+0000 7fc985400640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 0x7fc968071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:19.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.126+0000 7fc985400640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 0x7fc968071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc988159a30 tx=0x7fc974009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:19.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.130+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc97805be80 con 0x7fc98806c790 2024-05-27T06:53:19.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.386+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc98815bf40 con 0x7fc98806c790 2024-05-27T06:53:19.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc97e000640 1 -- 172.21.15.81:0/2633051844 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6082 (secure 0 0 0) 0x7fc978060180 con 0x7fc98806c790 2024-05-27T06:53:19.381 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:19.381 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:14.089849+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:19.382 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 38 2024-05-27T06:53:19.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 msgr2=0x7fc968071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 0x7fc968071910 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc988159a30 tx=0x7fc974009040 comp rx=0 tx=0).stop 2024-05-27T06:53:19.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 msgr2=0x7fc988154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc97800d5f0 tx=0x7fc97800dab0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 shutdown_connections 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc9880c8f90 0x7fc9881491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc96806f400 0x7fc968071910 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc98815cb80 0x7fc98815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 --2- 172.21.15.81:0/2633051844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc98806c790 0x7fc988154f90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 >> 172.21.15.81:0/2633051844 conn(0x7fc988070700 msgr2=0x7fc988151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 shutdown_connections 2024-05-27T06:53:19.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:19.390+0000 7fc98ce00640 1 -- 172.21.15.81:0/2633051844 wait complete. 2024-05-27T06:53:19.403 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:21.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:21.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.574+0000 7fcc5f000640 1 Processor -- start 2024-05-27T06:53:21.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.574+0000 7fcc5f000640 1 -- start start 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5f000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcc581546f0 0x7fcc580c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5f000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcc580c7bf0 0x7fcc58164dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5f000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc5806d0a0 con 0x7fcc58072d10 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5f000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fcc5806d620 con 0x7fcc580c7bf0 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5f000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fcc58154b00 con 0x7fcc581546f0 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcc580c7bf0 0x7fcc58164dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcc580c7bf0 0x7fcc58164dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50688/0 (socket says 172.21.15.81:50688) 2024-05-27T06:53:21.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 -- 172.21.15.81:0/287148719 learned_addr learned my addr 172.21.15.81:0/287148719 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:21.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 -- 172.21.15.81:0/287148719 >> v2:172.21.15.106:3300/0 conn(0x7fcc581546f0 msgr2=0x7fcc580c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:21.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 --2- 172.21.15.81:0/287148719 >> v2:172.21.15.106:3300/0 conn(0x7fcc581546f0 0x7fcc580c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 -- 172.21.15.81:0/287148719 >> v1:172.21.15.106:6789/0 conn(0x7fcc58072d10 legacy=0x7fcc5815da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:21.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.578+0000 7fcc5d000640 1 -- 172.21.15.81:0/287148719 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5806c8a0 con 0x7fcc580c7bf0 2024-05-27T06:53:21.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5d000640 1 --2- 172.21.15.81:0/287148719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc580c7bf0 0x7fcc58164dc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcc58072ca0 tx=0x7fcc5001f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f8294044d8f5bd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:21.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc56a00640 1 -- 172.21.15.81:0/287148719 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc500b4070 con 0x7fcc580c7bf0 2024-05-27T06:53:21.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc56a00640 1 -- 172.21.15.81:0/287148719 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc500af400 con 0x7fcc580c7bf0 2024-05-27T06:53:21.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc56a00640 1 -- 172.21.15.81:0/287148719 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc500af7d0 con 0x7fcc580c7bf0 2024-05-27T06:53:21.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 -- 172.21.15.81:0/287148719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc580c7bf0 msgr2=0x7fcc58164dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:21.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 --2- 172.21.15.81:0/287148719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc580c7bf0 0x7fcc58164dc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fcc58072ca0 tx=0x7fcc5001f1b0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 -- 172.21.15.81:0/287148719 shutdown_connections 2024-05-27T06:53:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 --2- 172.21.15.81:0/287148719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc580c7bf0 0x7fcc58164dc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 --2- 172.21.15.81:0/287148719 >> v2:172.21.15.106:3300/0 conn(0x7fcc581546f0 0x7fcc580c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 -- 172.21.15.81:0/287148719 >> 172.21.15.81:0/287148719 conn(0x7fcc58075060 msgr2=0x7fcc58070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 -- 172.21.15.81:0/287148719 shutdown_connections 2024-05-27T06:53:21.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.582+0000 7fcc5f000640 1 -- 172.21.15.81:0/287148719 wait complete. 2024-05-27T06:53:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 Processor -- start 2024-05-27T06:53:21.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 -- start start 2024-05-27T06:53:21.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc580c7bf0 0x7fcc58150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 0x7fcc58143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5806cb80 con 0x7fcc58072d10 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcc580ca200 con 0x7fcc581546f0 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.586+0000 7fcc5f000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5815e140 con 0x7fcc580c7bf0 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50704/0 (socket says 172.21.15.81:50704) 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 -- 172.21.15.81:0/1775271950 learned_addr learned my addr 172.21.15.81:0/1775271950 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc5d000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 0x7fcc58143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:21.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57400640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc580c7bf0 0x7fcc58150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:21.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 -- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc580c7bf0 msgr2=0x7fcc58150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:21.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc580c7bf0 0x7fcc58150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 -- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 msgr2=0x7fcc58143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 0x7fcc58143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 -- 172.21.15.81:0/1775271950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5814a820 con 0x7fcc58072d10 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc5d000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 0x7fcc58143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc57e00640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b410 tx=0x7fcc4800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48015030 con 0x7fcc58072d10 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc48004020 con 0x7fcc58072d10 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc58148fb0 con 0x7fcc58072d10 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48004450 con 0x7fcc58072d10 2024-05-27T06:53:21.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5815dbb0 con 0x7fcc58072d10 2024-05-27T06:53:21.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.590+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc58148da0 con 0x7fcc58072d10 2024-05-27T06:53:21.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.594+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcc48018070 con 0x7fcc58072d10 2024-05-27T06:53:21.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.594+0000 7fcc4fe00640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 0x7fcc38071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:21.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.594+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7fcc4808edf0 con 0x7fcc58072d10 2024-05-27T06:53:21.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.594+0000 7fcc57400640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 0x7fcc38071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:21.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.594+0000 7fcc57400640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 0x7fcc38071940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc58147fb0 tx=0x7fcc44006f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:21.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.602+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fcc4805bc30 con 0x7fcc58072d10 2024-05-27T06:53:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.838+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc5806a560 con 0x7fcc58072d10 2024-05-27T06:53:21.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.838+0000 7fcc4fe00640 1 -- 172.21.15.81:0/1775271950 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6079 (secure 0 0 0) 0x7fcc4805b530 con 0x7fcc58072d10 2024-05-27T06:53:21.831 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:21.831 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:19.684534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,4287]},"id":1}]} 2024-05-27T06:53:21.831 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 39 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 msgr2=0x7fcc38071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 0x7fcc38071940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc58147fb0 tx=0x7fcc44006f90 comp rx=0 tx=0).stop 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 msgr2=0x7fcc5814ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b410 tx=0x7fcc4800b8d0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 shutdown_connections 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcc580c7bf0 0x7fcc58150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcc3806f430 0x7fcc38071940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc581546f0 0x7fcc58143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 --2- 172.21.15.81:0/1775271950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58072d10 0x7fcc5814ebf0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:21.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 >> 172.21.15.81:0/1775271950 conn(0x7fcc58075060 msgr2=0x7fcc580cd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:21.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 shutdown_connections 2024-05-27T06:53:21.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:21.842+0000 7fcc5f000640 1 -- 172.21.15.81:0/1775271950 wait complete. 2024-05-27T06:53:21.856 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:23.857 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:24.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 Processor -- start 2024-05-27T06:53:24.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 -- start start 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f18a0072700 0x7f18a00730d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a006d5b0 con 0x7f18a00c9990 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a0074ca0 con 0x7f18a00c9580 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f18a0075540 con 0x7f18a0072700 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f189fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f18a0072700 0x7f18a00730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f18a00c9990 0x7f18a006cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44644/0 (socket says 172.21.15.81:44644) 2024-05-27T06:53:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f18a5000640 1 -- 172.21.15.81:0/4073417282 learned_addr learned my addr 172.21.15.81:0/4073417282 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f189fe00640 1 -- 172.21.15.81:0/4073417282 >> v1:172.21.15.106:6789/0 conn(0x7f18a00c9580 legacy=0x7f18a006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.030+0000 7f189ea00640 1 -- 172.21.15.81:0/4073417282 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012260602 0 0) 0x7f18a006d5b0 con 0x7f18a00c9990 2024-05-27T06:53:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189fe00640 1 -- 172.21.15.81:0/4073417282 >> v1:172.21.15.81:6789/0 conn(0x7f18a00c9990 legacy=0x7f18a006cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189fe00640 1 -- 172.21.15.81:0/4073417282 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a0074f80 con 0x7f18a0072700 2024-05-27T06:53:24.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189fe00640 1 --2- 172.21.15.81:0/4073417282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a00730d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f18900097c0 tx=0x7f189001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfee03d8e6f5afa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189ea00640 1 -- 172.21.15.81:0/4073417282 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18900b5070 con 0x7f18a0072700 2024-05-27T06:53:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189ea00640 1 -- 172.21.15.81:0/4073417282 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18900a8990 con 0x7f18a0072700 2024-05-27T06:53:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.034+0000 7f189ea00640 1 -- 172.21.15.81:0/4073417282 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18900a8d60 con 0x7f18a0072700 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- 172.21.15.81:0/4073417282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 msgr2=0x7f18a00730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 --2- 172.21.15.81:0/4073417282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a00730d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f18900097c0 tx=0x7f189001ec90 comp rx=0 tx=0).stop 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- 172.21.15.81:0/4073417282 shutdown_connections 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 --2- 172.21.15.81:0/4073417282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a00730d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- 172.21.15.81:0/4073417282 >> 172.21.15.81:0/4073417282 conn(0x7f18a00ce540 msgr2=0x7f18a00701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- 172.21.15.81:0/4073417282 shutdown_connections 2024-05-27T06:53:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- 172.21.15.81:0/4073417282 wait complete. 2024-05-27T06:53:24.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 Processor -- start 2024-05-27T06:53:24.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.038+0000 7f18a5a00640 1 -- start start 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18a00c9580 0x7f18a014f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18a00c9990 0x7f18a01454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f18a00c6b50 con 0x7f18a0072700 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f18a0074a50 con 0x7f18a00c9990 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f18a00c67e0 con 0x7f18a00c9580 2024-05-27T06:53:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50726/0 (socket says 172.21.15.81:50726) 2024-05-27T06:53:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 -- 172.21.15.81:0/1097770601 learned_addr learned my addr 172.21.15.81:0/1097770601 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5000640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18a00c9990 0x7f18a01454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189f400640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18a00c9580 0x7f18a014f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 -- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18a00c9580 msgr2=0x7f18a014f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18a00c9580 0x7f18a014f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 -- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18a00c9990 msgr2=0x7f18a01454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18a00c9990 0x7f18a01454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 -- 172.21.15.81:0/1097770601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a01f4910 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f189fe00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f18900097c0 tx=0x7f189001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18900c0070 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1890004230 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18900b5040 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a01f1300 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a01f4b80 con 0x7f18a0072700 2024-05-27T06:53:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.042+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a0050870 con 0x7f18a0072700 2024-05-27T06:53:24.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.046+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f18900c3070 con 0x7f18a0072700 2024-05-27T06:53:24.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.046+0000 7f1897e00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 0x7f1880071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:24.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.046+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f189012be30 con 0x7f18a0072700 2024-05-27T06:53:24.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.046+0000 7f189f400640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 0x7f1880071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:24.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.046+0000 7f189f400640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 0x7f1880071940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f18a0150ad0 tx=0x7f188c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:24.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.050+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f18900187f0 con 0x7f18a0072700 2024-05-27T06:53:24.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.298+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18a0150b00 con 0x7f18a0072700 2024-05-27T06:53:24.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.298+0000 7f1897e00640 1 -- 172.21.15.81:0/1097770601 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6079 (secure 0 0 0) 0x7f18900f8470 con 0x7f18a0072700 2024-05-27T06:53:24.291 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:24.291 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:19.684534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,4287]},"id":1}]} 2024-05-27T06:53:24.291 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 39 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 msgr2=0x7f1880071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 0x7f1880071940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f18a0150ad0 tx=0x7f188c009040 comp rx=0 tx=0).stop 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 msgr2=0x7f18a01f5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f18900097c0 tx=0x7f189001ba30 comp rx=0 tx=0).stop 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 shutdown_connections 2024-05-27T06:53:24.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f18a00c9580 0x7f18a014f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f188006f430 0x7f1880071940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18a00c9990 0x7f18a01454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 --2- 172.21.15.81:0/1097770601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18a0072700 0x7f18a01f5380 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 >> 172.21.15.81:0/1097770601 conn(0x7f18a00ce540 msgr2=0x7f18a00ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 shutdown_connections 2024-05-27T06:53:24.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:24.302+0000 7f18a5a00640 1 -- 172.21.15.81:0/1097770601 wait complete. 2024-05-27T06:53:24.313 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:53:26.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:26.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.486+0000 7f1d7de00640 1 Processor -- start 2024-05-27T06:53:26.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.486+0000 7f1d7de00640 1 -- start start 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 0x7f1d78165170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d780ca8a0 con 0x7f1d780cb590 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7de00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d780733a0 con 0x7f1d78156ac0 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1d78156ed0 con 0x7f1d78162d50 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1d780cb590 0x7f1d780cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38540/0 (socket says 172.21.15.81:38540) 2024-05-27T06:53:26.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77e00640 1 -- 172.21.15.81:0/3092997971 learned_addr learned my addr 172.21.15.81:0/3092997971 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7d400640 1 --2- 172.21.15.81:0/3092997971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 0x7f1d78165170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2132081053 0 0) 0x7f1d780ca8a0 con 0x7f1d780cb590 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d60003530 con 0x7f1d780cb590 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694932071 0 0) 0x7f1d780733a0 con 0x7f1d78156ac0 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d780ca8a0 con 0x7f1d78156ac0 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7d400640 1 -- 172.21.15.81:0/3092997971 >> v1:172.21.15.106:6789/0 conn(0x7f1d78156ac0 legacy=0x7f1d780cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7d400640 1 -- 172.21.15.81:0/3092997971 >> v1:172.21.15.81:6789/0 conn(0x7f1d780cb590 legacy=0x7f1d780cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.490+0000 7f1d7d400640 1 -- 172.21.15.81:0/3092997971 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d78073640 con 0x7f1d78162d50 2024-05-27T06:53:26.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7d400640 1 --2- 172.21.15.81:0/3092997971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 0x7f1d78165170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009dc0 tx=0x7f1d7001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e2d8174a283b62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700b2070 con 0x7f1d78162d50 2024-05-27T06:53:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d700a5850 con 0x7f1d78162d50 2024-05-27T06:53:26.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d77400640 1 -- 172.21.15.81:0/3092997971 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700a5c00 con 0x7f1d78162d50 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 -- 172.21.15.81:0/3092997971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 msgr2=0x7f1d78165170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 --2- 172.21.15.81:0/3092997971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 0x7f1d78165170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009dc0 tx=0x7f1d7001c990 comp rx=0 tx=0).stop 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 -- 172.21.15.81:0/3092997971 shutdown_connections 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 --2- 172.21.15.81:0/3092997971 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78162d50 0x7f1d78165170 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 -- 172.21.15.81:0/3092997971 >> 172.21.15.81:0/3092997971 conn(0x7f1d78071390 msgr2=0x7f1d780717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 -- 172.21.15.81:0/3092997971 shutdown_connections 2024-05-27T06:53:26.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.494+0000 7f1d7de00640 1 -- 172.21.15.81:0/3092997971 wait complete. 2024-05-27T06:53:26.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 Processor -- start 2024-05-27T06:53:26.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 -- start start 2024-05-27T06:53:26.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d780cb590 0x7f1d781f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:26.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78156ac0 0x7f1d780cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:26.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d780cf3f0 con 0x7f1d78162d50 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1d780cb070 con 0x7f1d780cb590 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.498+0000 7f1d7de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1d780cf570 con 0x7f1d78156ac0 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44358/0 (socket says 172.21.15.81:44358) 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 -- 172.21.15.81:0/197480902 learned_addr learned my addr 172.21.15.81:0/197480902 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d77e00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78156ac0 0x7f1d780cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:26.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7ca00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d780cb590 0x7f1d781f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 -- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78156ac0 msgr2=0x7f1d780cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78156ac0 0x7f1d780cd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 -- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d780cb590 msgr2=0x7f1d781f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d780cb590 0x7f1d781f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 -- 172.21.15.81:0/197480902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d781490f0 con 0x7f1d78162d50 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7d400640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1d700b0040 tx=0x7f1d70019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700b2050 con 0x7f1d78162d50 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d700be070 con 0x7f1d78162d50 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d700ad890 con 0x7f1d78162d50 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d781f68f0 con 0x7f1d78162d50 2024-05-27T06:53:26.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d7815c730 con 0x7f1d78162d50 2024-05-27T06:53:26.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d780caad0 con 0x7f1d78162d50 2024-05-27T06:53:26.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.502+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1d700ada30 con 0x7f1d78162d50 2024-05-27T06:53:26.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.506+0000 7f1d74c00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 0x7f1d58071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:26.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.506+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f1d7012a230 con 0x7f1d78162d50 2024-05-27T06:53:26.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.506+0000 7f1d7ca00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 0x7f1d58071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:26.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.506+0000 7f1d7ca00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 0x7f1d58071910 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c007fd0 tx=0x7f1d6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:26.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.510+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1d70016580 con 0x7f1d78162d50 2024-05-27T06:53:26.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.758+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d780c74c0 con 0x7f1d78162d50 2024-05-27T06:53:26.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d74c00640 1 -- 172.21.15.81:0/197480902 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6084 (secure 0 0 0) 0x7f1d700f6a20 con 0x7f1d78162d50 2024-05-27T06:53:26.753 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:26.753 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,5184,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:26.753 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 40 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 msgr2=0x7f1d58071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 0x7f1d58071910 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c007fd0 tx=0x7f1d6c009040 comp rx=0 tx=0).stop 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 msgr2=0x7f1d78144980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1d700b0040 tx=0x7f1d70019730 comp rx=0 tx=0).stop 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 shutdown_connections 2024-05-27T06:53:26.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1d78156ac0 0x7f1d780cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1d5806f400 0x7f1d58071910 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d78162d50 0x7f1d78144980 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 --2- 172.21.15.81:0/197480902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1d780cb590 0x7f1d781f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 >> 172.21.15.81:0/197480902 conn(0x7f1d78071390 msgr2=0x7f1d780cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 shutdown_connections 2024-05-27T06:53:26.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:26.762+0000 7f1d7de00640 1 -- 172.21.15.81:0/197480902 wait complete. 2024-05-27T06:53:26.777 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4285 2024-05-27T06:53:26.777 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2024-05-27T06:53:34.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:53:34.155 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:53:35.779 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-05-27T06:53:35.779 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-27T06:53:35.779 DEBUG:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-27T06:53:35.784 INFO:tasks.ceph.mds.b:Started 2024-05-27T06:53:35.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:35.889 INFO:tasks.ceph.mds.b.smithi106.stdout:starting mds.b at 2024-05-27T06:53:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.970+0000 7f9d6ac00640 1 Processor -- start 2024-05-27T06:53:35.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 -- start start 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d640ca240 0x7f9d6415c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9d640c96e0 0x7f9d640c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6406c9a0 con 0x7f9d64156b40 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6406db10 con 0x7f9d640c96e0 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d6ac00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6406cc40 con 0x7f9d640ca240 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d63400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9d64156b40 0x7f9d640cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51404/0 (socket says 172.21.15.81:51404) 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d63400640 1 -- 172.21.15.81:0/2319731403 learned_addr learned my addr 172.21.15.81:0/2319731403 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:35.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d68c00640 1 --2- 172.21.15.81:0/2319731403 >> v2:172.21.15.81:3300/0 conn(0x7f9d640c96e0 0x7f9d640c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d63e00640 1 --2- 172.21.15.81:0/2319731403 >> v2:172.21.15.106:3300/0 conn(0x7f9d640ca240 0x7f9d6415c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d62a00640 1 -- 172.21.15.81:0/2319731403 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252462945 0 0) 0x7f9d6406c9a0 con 0x7f9d64156b40 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d62a00640 1 -- 172.21.15.81:0/2319731403 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d48003530 con 0x7f9d64156b40 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d68c00640 1 -- 172.21.15.81:0/2319731403 >> v2:172.21.15.106:3300/0 conn(0x7f9d640ca240 msgr2=0x7f9d6415c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d68c00640 1 --2- 172.21.15.81:0/2319731403 >> v2:172.21.15.106:3300/0 conn(0x7f9d640ca240 0x7f9d6415c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d68c00640 1 -- 172.21.15.81:0/2319731403 >> v1:172.21.15.81:6789/0 conn(0x7f9d64156b40 legacy=0x7f9d640cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:35.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.974+0000 7f9d68c00640 1 -- 172.21.15.81:0/2319731403 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6406cf00 con 0x7f9d640c96e0 2024-05-27T06:53:35.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d68c00640 1 --2- 172.21.15.81:0/2319731403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d640c96e0 0x7f9d640c9af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ca210 tx=0x7f9d5c01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599b3d1dd727672f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:35.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d62a00640 1 -- 172.21.15.81:0/2319731403 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0b4070 con 0x7f9d640c96e0 2024-05-27T06:53:35.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d62a00640 1 -- 172.21.15.81:0/2319731403 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5c0af600 con 0x7f9d640c96e0 2024-05-27T06:53:35.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d62a00640 1 -- 172.21.15.81:0/2319731403 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0af9b0 con 0x7f9d640c96e0 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 -- 172.21.15.81:0/2319731403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d640c96e0 msgr2=0x7f9d640c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/2319731403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d640c96e0 0x7f9d640c9af0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ca210 tx=0x7f9d5c01f110 comp rx=0 tx=0).stop 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 -- 172.21.15.81:0/2319731403 shutdown_connections 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/2319731403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d640c96e0 0x7f9d640c9af0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/2319731403 >> v2:172.21.15.106:3300/0 conn(0x7f9d640ca240 0x7f9d6415c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 -- 172.21.15.81:0/2319731403 >> 172.21.15.81:0/2319731403 conn(0x7f9d640cb2c0 msgr2=0x7f9d640cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:35.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 -- 172.21.15.81:0/2319731403 shutdown_connections 2024-05-27T06:53:35.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.978+0000 7f9d6ac00640 1 -- 172.21.15.81:0/2319731403 wait complete. 2024-05-27T06:53:35.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.982+0000 7f9d6ac00640 1 Processor -- start 2024-05-27T06:53:35.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.982+0000 7f9d6ac00640 1 -- start start 2024-05-27T06:53:35.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d640c96e0 0x7f9d6415eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d640ca240 0x7f9d641f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6406d7a0 con 0x7f9d64156b40 2024-05-27T06:53:35.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6406d4c0 con 0x7f9d640c96e0 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6406dfa0 con 0x7f9d640ca240 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39636/0 (socket says 172.21.15.81:39636) 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 -- 172.21.15.81:0/3942074829 learned_addr learned my addr 172.21.15.81:0/3942074829 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d63e00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d640c96e0 0x7f9d6415eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d63400640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d640ca240 0x7f9d641f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 -- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d640ca240 msgr2=0x7f9d641f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:35.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d640ca240 0x7f9d641f6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:35.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 -- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d640c96e0 msgr2=0x7f9d6415eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:35.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d640c96e0 0x7f9d6415eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:35.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 -- 172.21.15.81:0/3942074829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6415f070 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d68c00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0b2040 tx=0x7f9d5c01beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c0b4040 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5c0039a0 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d641e8580 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c009450 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6415e330 con 0x7f9d64156b40 2024-05-27T06:53:35.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.986+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6406a560 con 0x7f9d64156b40 2024-05-27T06:53:35.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.990+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9d5c004030 con 0x7f9d64156b40 2024-05-27T06:53:35.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.990+0000 7f9d5be00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:35.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.990+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f9d5c12d480 con 0x7f9d64156b40 2024-05-27T06:53:35.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.990+0000 7f9d63e00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:35.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.990+0000 7f9d63e00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 0x7f9d44071910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009ac0 tx=0x7f9d54009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:35.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:35.994+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9d5c018c70 con 0x7f9d64156b40 2024-05-27T06:53:36.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.238+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d640cb030 con 0x7f9d64156b40 2024-05-27T06:53:36.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.238+0000 7f9d5be00640 1 -- 172.21.15.81:0/3942074829 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6084 (secure 0 0 0) 0x7f9d5c0f9ab0 con 0x7f9d64156b40 2024-05-27T06:53:36.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:36.232 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4287,5184,4302,4870,5049]},"id":1}]} 2024-05-27T06:53:36.232 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 40 2024-05-27T06:53:36.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 msgr2=0x7f9d44071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:36.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 0x7f9d44071910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009ac0 tx=0x7f9d54009190 comp rx=0 tx=0).stop 2024-05-27T06:53:36.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 msgr2=0x7f9d641f6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:36.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0b2040 tx=0x7f9d5c01beb0 comp rx=0 tx=0).stop 2024-05-27T06:53:36.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 shutdown_connections 2024-05-27T06:53:36.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d640ca240 0x7f9d641f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d4406f400 0x7f9d44071910 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d64156b40 0x7f9d641f6980 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 --2- 172.21.15.81:0/3942074829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d640c96e0 0x7f9d6415eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 >> 172.21.15.81:0/3942074829 conn(0x7f9d640cb2c0 msgr2=0x7f9d64150510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 shutdown_connections 2024-05-27T06:53:36.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:36.242+0000 7f9d6ac00640 1 -- 172.21.15.81:0/3942074829 wait complete. 2024-05-27T06:53:36.259 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-05-27T06:53:38.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:38.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 Processor -- start 2024-05-27T06:53:38.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 -- start start 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47681549b0 0x7f476815dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f476815e150 0x7f47680c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4768073910 con 0x7f47680c7240 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f47680c9ba0 con 0x7f476815e150 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476e600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4768073630 con 0x7f47681549b0 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f4767e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f476815e150 0x7f47680c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f4767e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f476815e150 0x7f47680c6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39648/0 (socket says 172.21.15.81:39648) 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f4767e00640 1 -- 172.21.15.81:0/2570344423 learned_addr learned my addr 172.21.15.81:0/2570344423 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:38.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.434+0000 7f476d200640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47681549b0 0x7f476815dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767e00640 1 -- 172.21.15.81:0/2570344423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47681549b0 msgr2=0x7f476815dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:38.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767e00640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47681549b0 0x7f476815dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767e00640 1 -- 172.21.15.81:0/2570344423 >> v1:172.21.15.106:6789/0 conn(0x7f47680c7240 legacy=0x7f4768072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767e00640 1 -- 172.21.15.81:0/2570344423 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680c9270 con 0x7f476815e150 2024-05-27T06:53:38.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767e00640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47680c6d00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4754009870 tx=0x7f475401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7512646c40ed2e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:38.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767400640 1 -- 172.21.15.81:0/2570344423 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47540b2070 con 0x7f476815e150 2024-05-27T06:53:38.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767400640 1 -- 172.21.15.81:0/2570344423 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47540a9070 con 0x7f476815e150 2024-05-27T06:53:38.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.438+0000 7f4767400640 1 -- 172.21.15.81:0/2570344423 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47540ad880 con 0x7f476815e150 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- 172.21.15.81:0/2570344423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 msgr2=0x7f47680c6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47680c6d00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4754009870 tx=0x7f475401c560 comp rx=0 tx=0).stop 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- 172.21.15.81:0/2570344423 shutdown_connections 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47680c6d00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 --2- 172.21.15.81:0/2570344423 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47681549b0 0x7f476815dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- 172.21.15.81:0/2570344423 >> 172.21.15.81:0/2570344423 conn(0x7f476806d180 msgr2=0x7f47680cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- 172.21.15.81:0/2570344423 shutdown_connections 2024-05-27T06:53:38.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- 172.21.15.81:0/2570344423 wait complete. 2024-05-27T06:53:38.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 Processor -- start 2024-05-27T06:53:38.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.442+0000 7f476e600640 1 -- start start 2024-05-27T06:53:38.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47680c7240 0x7f47680cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47681549b0 0x7f47680cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4768167b50 con 0x7f476815e150 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f47681679d0 con 0x7f47681549b0 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4768167cd0 con 0x7f47680c7240 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39660/0 (socket says 172.21.15.81:39660) 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 -- 172.21.15.81:0/3348017084 learned_addr learned my addr 172.21.15.81:0/3348017084 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f4767e00640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47681549b0 0x7f47680cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476d200640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47680c7240 0x7f47680cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 -- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47680c7240 msgr2=0x7f47680cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47680c7240 0x7f47680cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 -- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47681549b0 msgr2=0x7f47680cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47681549b0 0x7f47680cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 -- 172.21.15.81:0/3348017084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680cce50 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476dc00640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f476000c760 tx=0x7f476000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4760003b80 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4760003d20 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47680cbb50 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476000aa70 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47680cc110 con 0x7f476815e150 2024-05-27T06:53:38.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.446+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4768050870 con 0x7f476815e150 2024-05-27T06:53:38.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.450+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f476000ac30 con 0x7f476815e150 2024-05-27T06:53:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.450+0000 7f4764c00640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 0x7f4748071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.450+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f476008f230 con 0x7f476815e150 2024-05-27T06:53:38.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.450+0000 7f476d200640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 0x7f4748071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:38.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.450+0000 7f476d200640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 0x7f4748071910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f475c009ac0 tx=0x7f475c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:38.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.454+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4768073910 con 0x7f476815e150 2024-05-27T06:53:38.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4768152930 con 0x7f476815e150 2024-05-27T06:53:38.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f4764c00640 1 -- 172.21.15.81:0/3348017084 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f476005b8f0 con 0x7f476815e150 2024-05-27T06:53:38.685 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:38.686 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:38.686 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 msgr2=0x7f4748071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 0x7f4748071910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f475c009ac0 tx=0x7f475c009190 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 msgr2=0x7f47681498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f476000c760 tx=0x7f476000cc20 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 shutdown_connections 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f474806f400 0x7f4748071910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f476815e150 0x7f47681498d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f47681549b0 0x7f47680cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 --2- 172.21.15.81:0/3348017084 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f47680c7240 0x7f47680cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:38.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 >> 172.21.15.81:0/3348017084 conn(0x7f476806d180 msgr2=0x7f476814f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:38.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 shutdown_connections 2024-05-27T06:53:38.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:38.694+0000 7f476e600640 1 -- 172.21.15.81:0/3348017084 wait complete. 2024-05-27T06:53:38.709 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-05-27T06:53:38.709 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2024-05-27T06:53:40.710 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:40.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 Processor -- start 2024-05-27T06:53:40.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 -- start start 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8700736c0 0x7fe870073c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe870075010 0x7fe870167040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87015d710 con 0x7fe870074c00 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe87006c8c0 con 0x7fe870075010 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe876e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe870075420 con 0x7fe8700736c0 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe874e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe870075010 0x7fe870167040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe874e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe870075010 0x7fe870167040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:44920/0 (socket says 172.21.15.81:44920) 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe874e00640 1 -- 172.21.15.81:0/2194297645 learned_addr learned my addr 172.21.15.81:0/2194297645 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:40.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.886+0000 7fe86fe00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8700736c0 0x7fe870073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe874e00640 1 -- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8700736c0 msgr2=0x7fe870073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:40.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe874e00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8700736c0 0x7fe870073c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:40.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe874e00640 1 -- 172.21.15.81:0/2194297645 >> v1:172.21.15.106:6789/0 conn(0x7fe870074c00 legacy=0x7fe87006d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:40.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe874e00640 1 -- 172.21.15.81:0/2194297645 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87006cb60 con 0x7fe870075010 2024-05-27T06:53:40.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe86f400640 1 -- 172.21.15.81:0/2194297645 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553292006 0 0) 0x7fe87015d710 con 0x7fe870074c00 2024-05-27T06:53:40.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe874e00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe870075010 0x7fe870167040 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe868009e50 tx=0x7fe86801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ae380b0d09f3a67 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:40.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe86f400640 1 -- 172.21.15.81:0/2194297645 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8680a5700 con 0x7fe870075010 2024-05-27T06:53:40.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.890+0000 7fe86f400640 1 -- 172.21.15.81:0/2194297645 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8680a58a0 con 0x7fe870075010 2024-05-27T06:53:40.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe86f400640 1 -- 172.21.15.81:0/2194297645 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8680a5c70 con 0x7fe870075010 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe870075010 msgr2=0x7fe870167040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe870075010 0x7fe870167040 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe868009e50 tx=0x7fe86801ca30 comp rx=0 tx=0).stop 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- 172.21.15.81:0/2194297645 shutdown_connections 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe870075010 0x7fe870167040 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 --2- 172.21.15.81:0/2194297645 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8700736c0 0x7fe870073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- 172.21.15.81:0/2194297645 >> 172.21.15.81:0/2194297645 conn(0x7fe8700cefd0 msgr2=0x7fe8700cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- 172.21.15.81:0/2194297645 shutdown_connections 2024-05-27T06:53:40.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- 172.21.15.81:0/2194297645 wait complete. 2024-05-27T06:53:40.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 Processor -- start 2024-05-27T06:53:40.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.894+0000 7fe876e00640 1 -- start start 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8700736c0 0x7fe8700c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe870075010 0x7fe870147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe870156f50 con 0x7fe870074c00 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe870156dd0 con 0x7fe8700736c0 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe87006d830 con 0x7fe870075010 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39662/0 (socket says 172.21.15.81:39662) 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 -- 172.21.15.81:0/3764088705 learned_addr learned my addr 172.21.15.81:0/3764088705 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe86fe00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8700736c0 0x7fe8700c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe874e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe870075010 0x7fe870147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 -- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe870075010 msgr2=0x7fe870147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:40.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe870075010 0x7fe870147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 -- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8700736c0 msgr2=0x7fe8700c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8700736c0 0x7fe8700c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 -- 172.21.15.81:0/3764088705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87014ba10 con 0x7fe870074c00 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe867e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c430 tx=0x7fe85800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858017070 con 0x7fe870074c00 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858003600 con 0x7fe870074c00 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8580039f0 con 0x7fe870074c00 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8701527a0 con 0x7fe870074c00 2024-05-27T06:53:40.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe870151700 con 0x7fe870074c00 2024-05-27T06:53:40.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.898+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8701480d0 con 0x7fe870074c00 2024-05-27T06:53:40.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.902+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe85801a070 con 0x7fe870074c00 2024-05-27T06:53:40.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.902+0000 7fe86cc00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 0x7fe84c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:40.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.902+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7fe85808e670 con 0x7fe870074c00 2024-05-27T06:53:40.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.902+0000 7fe86fe00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 0x7fe84c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:40.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.902+0000 7fe86fe00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 0x7fe84c071910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe860009ac0 tx=0x7fe860009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:40.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:40.906+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe85805ae60 con 0x7fe870074c00 2024-05-27T06:53:41.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.130+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8700c58d0 con 0x7fe870074c00 2024-05-27T06:53:41.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.130+0000 7fe86cc00640 1 -- 172.21.15.81:0/3764088705 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7fe85805f160 con 0x7fe870074c00 2024-05-27T06:53:41.123 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:41.123 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:41.123 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 msgr2=0x7fe84c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 0x7fe84c071910 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe860009ac0 tx=0x7fe860009190 comp rx=0 tx=0).stop 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 msgr2=0x7fe870147650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c430 tx=0x7fe85800c8f0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 shutdown_connections 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe870075010 0x7fe870147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe84c06f400 0x7fe84c071910 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe870074c00 0x7fe870147650 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 --2- 172.21.15.81:0/3764088705 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8700736c0 0x7fe8700c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 >> 172.21.15.81:0/3764088705 conn(0x7fe8700cefd0 msgr2=0x7fe8701549c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:41.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 shutdown_connections 2024-05-27T06:53:41.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.134+0000 7fe876e00640 1 -- 172.21.15.81:0/3764088705 wait complete. 2024-05-27T06:53:41.146 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=1) 2024-05-27T06:53:41.146 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T06:53:41.146 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:53:41.162 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T06:53:41.162 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:41.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.386+0000 7f634a800640 1 Processor -- start 2024-05-27T06:53:41.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.386+0000 7f634a800640 1 -- start start 2024-05-27T06:53:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f634a800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 0x7f63440d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f634a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634415ca00 con 0x7f634415e0f0 2024-05-27T06:53:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f634a800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634415cce0 con 0x7f63440d2490 2024-05-27T06:53:41.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f634a800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f634415dc60 con 0x7f6344156be0 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f634415e0f0 0x7f634415e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51418/0 (socket says 172.21.15.81:51418) 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343e00640 1 -- 172.21.15.81:0/1236654144 learned_addr learned my addr 172.21.15.81:0/1236654144 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6342a00640 1 --1- 172.21.15.81:0/1236654144 >> v1:172.21.15.106:6789/0 conn(0x7f63440d2490 0x7f63440cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:46364/0 (socket says 172.21.15.81:46364) 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343400640 1 --2- 172.21.15.81:0/1236654144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 0x7f63440d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 667881632 0 0) 0x7f634415ca00 con 0x7f634415e0f0 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6328003530 con 0x7f634415e0f0 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513889080 0 0) 0x7f634415cce0 con 0x7f63440d2490 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f634415ca00 con 0x7f63440d2490 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343400640 1 -- 172.21.15.81:0/1236654144 >> v1:172.21.15.106:6789/0 conn(0x7f63440d2490 legacy=0x7f63440cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343400640 1 -- 172.21.15.81:0/1236654144 >> v1:172.21.15.81:6789/0 conn(0x7f634415e0f0 legacy=0x7f634415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.390+0000 7f6343400640 1 -- 172.21.15.81:0/1236654144 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634415cfc0 con 0x7f6344156be0 2024-05-27T06:53:41.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f6343400640 1 --2- 172.21.15.81:0/1236654144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 0x7f63440d1f50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63340097f0 tx=0x7f633401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=840f89349fb5d9ae server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63340b5070 con 0x7f6344156be0 2024-05-27T06:53:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63340a89d0 con 0x7f6344156be0 2024-05-27T06:53:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f6342000640 1 -- 172.21.15.81:0/1236654144 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63340a8da0 con 0x7f6344156be0 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 -- 172.21.15.81:0/1236654144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 msgr2=0x7f63440d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 --2- 172.21.15.81:0/1236654144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 0x7f63440d1f50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63340097f0 tx=0x7f633401ecd0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 -- 172.21.15.81:0/1236654144 shutdown_connections 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 --2- 172.21.15.81:0/1236654144 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6344156be0 0x7f63440d1f50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 -- 172.21.15.81:0/1236654144 >> 172.21.15.81:0/1236654144 conn(0x7f634406e8f0 msgr2=0x7f634406ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 -- 172.21.15.81:0/1236654144 shutdown_connections 2024-05-27T06:53:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.394+0000 7f634a800640 1 -- 172.21.15.81:0/1236654144 wait complete. 2024-05-27T06:53:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 Processor -- start 2024-05-27T06:53:41.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 -- start start 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 0x7f6344157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6344156be0 0x7f63441e8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63440cd850 con 0x7f634415e0f0 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f634415d580 con 0x7f6344156be0 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f634a800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f63440cd9d0 con 0x7f63440d2490 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39676/0 (socket says 172.21.15.81:39676) 2024-05-27T06:53:41.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 -- 172.21.15.81:0/831744224 learned_addr learned my addr 172.21.15.81:0/831744224 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:41.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343400640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 0x7f6344157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:41.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6342a00640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6344156be0 0x7f63441e8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 -- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 msgr2=0x7f6344157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 0x7f6344157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 -- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6344156be0 msgr2=0x7f63441e8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6344156be0 0x7f63441e8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 -- 172.21.15.81:0/831744224 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63441f3b70 con 0x7f634415e0f0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343400640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 0x7f6344157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f6343e00640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f633c00ca80 tx=0x7f633c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c00ac30 con 0x7f634415e0f0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.398+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633c00add0 con 0x7f634415e0f0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63441f2db0 con 0x7f634415e0f0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c012600 con 0x7f634415e0f0 2024-05-27T06:53:41.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63441f3370 con 0x7f634415e0f0 2024-05-27T06:53:41.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6344158390 con 0x7f634415e0f0 2024-05-27T06:53:41.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f633c0127a0 con 0x7f634415e0f0 2024-05-27T06:53:41.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f633b400640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 0x7f6324071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:41.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f633c08f960 con 0x7f634415e0f0 2024-05-27T06:53:41.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.402+0000 7f6343400640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 0x7f6324071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:41.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.406+0000 7f6343400640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 0x7f6324071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63340b2000 tx=0x7f63340aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:41.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.406+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f633c05bf90 con 0x7f634415e0f0 2024-05-27T06:53:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.678+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f634406a560 con 0x7f634415e0f0 2024-05-27T06:53:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.678+0000 7f633b400640 1 -- 172.21.15.81:0/831744224 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f633c060290 con 0x7f634415e0f0 2024-05-27T06:53:41.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:41.671 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 msgr2=0x7f6324071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 0x7f6324071910 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f63340b2000 tx=0x7f63340aa070 comp rx=0 tx=0).stop 2024-05-27T06:53:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 msgr2=0x7f6344159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f633c00ca80 tx=0x7f633c00cf40 comp rx=0 tx=0).stop 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 shutdown_connections 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f632406f400 0x7f6324071910 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f634415e0f0 0x7f6344159640 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6344156be0 0x7f63441e8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 --2- 172.21.15.81:0/831744224 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f63440d2490 0x7f6344157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 >> 172.21.15.81:0/831744224 conn(0x7f634406e8f0 msgr2=0x7f63440b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 shutdown_connections 2024-05-27T06:53:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:41.682+0000 7f634a800640 1 -- 172.21.15.81:0/831744224 wait complete. 2024-05-27T06:53:41.695 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:43.695 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:43.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.870+0000 7f7c2d200640 1 Processor -- start 2024-05-27T06:53:43.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.870+0000 7f7c2d200640 1 -- start start 2024-05-27T06:53:43.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c2d200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f7c280cb590 0x7f7c280cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c2d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c28161d40 0x7f7c28164160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c2d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c280ca8a0 con 0x7f7c28156ac0 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c2d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f7c28156ed0 con 0x7f7c28161d40 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c2d200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f7c280733a0 con 0x7f7c280cb590 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c27400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7c28156ac0 0x7f7c280cece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51432/0 (socket says 172.21.15.81:51432) 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c27400640 1 -- 172.21.15.81:0/3404392534 learned_addr learned my addr 172.21.15.81:0/3404392534 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c27e00640 1 --2- 172.21.15.81:0/3404392534 >> v2:172.21.15.81:3300/0 conn(0x7f7c28161d40 0x7f7c28164160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26a00640 1 --2- 172.21.15.81:0/3404392534 >> v2:172.21.15.106:3300/0 conn(0x7f7c280cb590 0x7f7c280cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26000640 1 -- 172.21.15.81:0/3404392534 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737932979 0 0) 0x7f7c280ca8a0 con 0x7f7c28156ac0 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26000640 1 -- 172.21.15.81:0/3404392534 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c0c003530 con 0x7f7c28156ac0 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26a00640 1 -- 172.21.15.81:0/3404392534 >> v2:172.21.15.81:3300/0 conn(0x7f7c28161d40 msgr2=0x7f7c28164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26a00640 1 --2- 172.21.15.81:0/3404392534 >> v2:172.21.15.81:3300/0 conn(0x7f7c28161d40 0x7f7c28164160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26a00640 1 -- 172.21.15.81:0/3404392534 >> v1:172.21.15.81:6789/0 conn(0x7f7c28156ac0 legacy=0x7f7c280cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:43.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.874+0000 7f7c26a00640 1 -- 172.21.15.81:0/3404392534 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c28073640 con 0x7f7c280cb590 2024-05-27T06:53:43.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c26a00640 1 --2- 172.21.15.81:0/3404392534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c280cb590 0x7f7c280cb9a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7c280c6a60 tx=0x7f7c1401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fba10e7026f59244 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:43.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c26000640 1 -- 172.21.15.81:0/3404392534 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c140b0070 con 0x7f7c280cb590 2024-05-27T06:53:43.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c26000640 1 -- 172.21.15.81:0/3404392534 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c140ab600 con 0x7f7c280cb590 2024-05-27T06:53:43.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c26000640 1 -- 172.21.15.81:0/3404392534 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c140ab9b0 con 0x7f7c280cb590 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 -- 172.21.15.81:0/3404392534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c280cb590 msgr2=0x7f7c280cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 --2- 172.21.15.81:0/3404392534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c280cb590 0x7f7c280cb9a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7c280c6a60 tx=0x7f7c1401c480 comp rx=0 tx=0).stop 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 -- 172.21.15.81:0/3404392534 shutdown_connections 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 --2- 172.21.15.81:0/3404392534 >> v2:172.21.15.81:3300/0 conn(0x7f7c28161d40 0x7f7c28164160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 --2- 172.21.15.81:0/3404392534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c280cb590 0x7f7c280cb9a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 -- 172.21.15.81:0/3404392534 >> 172.21.15.81:0/3404392534 conn(0x7f7c28071390 msgr2=0x7f7c280717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 -- 172.21.15.81:0/3404392534 shutdown_connections 2024-05-27T06:53:43.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.878+0000 7f7c2d200640 1 -- 172.21.15.81:0/3404392534 wait complete. 2024-05-27T06:53:43.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.882+0000 7f7c2d200640 1 Processor -- start 2024-05-27T06:53:43.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.882+0000 7f7c2d200640 1 -- start start 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c28156ac0 0x7f7c2814e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c28161d40 0x7f7c2814ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c280cbee0 con 0x7f7c280cb590 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7c280cb070 con 0x7f7c28161d40 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7c280cf3f0 con 0x7f7c28156ac0 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39694/0 (socket says 172.21.15.81:39694) 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 -- 172.21.15.81:0/2142767087 learned_addr learned my addr 172.21.15.81:0/2142767087 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27e00640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c28161d40 0x7f7c2814ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c26a00640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c28156ac0 0x7f7c2814e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 -- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c28156ac0 msgr2=0x7f7c2814e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:43.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c28156ac0 0x7f7c2814e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 -- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c28161d40 msgr2=0x7f7c2814ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c28161d40 0x7f7c2814ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 -- 172.21.15.81:0/2142767087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2815cc60 con 0x7f7c280cb590 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c27400640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c430 tx=0x7f7c1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18017070 con 0x7f7c280cb590 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18005cf0 con 0x7f7c280cb590 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2815c980 con 0x7f7c280cb590 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180072d0 con 0x7f7c280cb590 2024-05-27T06:53:43.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c28151210 con 0x7f7c280cb590 2024-05-27T06:53:43.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.886+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c28050870 con 0x7f7c280cb590 2024-05-27T06:53:43.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.890+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7c1801a070 con 0x7f7c280cb590 2024-05-27T06:53:43.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.890+0000 7f7c1f400640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 0x7f7c08071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:43.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.890+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f7c1808e780 con 0x7f7c280cb590 2024-05-27T06:53:43.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.890+0000 7f7c26a00640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 0x7f7c08071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:43.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.890+0000 7f7c26a00640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 0x7f7c08071940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7c140ad000 tx=0x7f7c140a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:43.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:43.894+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7c1805ae50 con 0x7f7c280cb590 2024-05-27T06:53:44.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.154+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c280c6e90 con 0x7f7c280cb590 2024-05-27T06:53:44.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c1f400640 1 -- 172.21.15.81:0/2142767087 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f7c1805f150 con 0x7f7c280cb590 2024-05-27T06:53:44.148 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:44.149 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:44.149 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 msgr2=0x7f7c08071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 0x7f7c08071940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7c140ad000 tx=0x7f7c140a6000 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 msgr2=0x7f7c28159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7c1800c430 tx=0x7f7c1800c8f0 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 shutdown_connections 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7c28156ac0 0x7f7c2814e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c0806f430 0x7f7c08071940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c28161d40 0x7f7c2814ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 --2- 172.21.15.81:0/2142767087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c280cb590 0x7f7c28159e00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 >> 172.21.15.81:0/2142767087 conn(0x7f7c28071390 msgr2=0x7f7c28073dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 shutdown_connections 2024-05-27T06:53:44.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:44.158+0000 7f7c2d200640 1 -- 172.21.15.81:0/2142767087 wait complete. 2024-05-27T06:53:44.173 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:46.174 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.358+0000 7f1774000640 1 Processor -- start 2024-05-27T06:53:46.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.358+0000 7f1774000640 1 -- start start 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f176c154970 0x7f176c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c15daf0 0x7f176c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f176c0c8410 0x7f176c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f176c06eec0 con 0x7f176c154970 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f176c06f0f0 con 0x7f176c0c8410 2024-05-27T06:53:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1774000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f176c06f3b0 con 0x7f176c15daf0 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f176c154970 0x7f176c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f176c154970 0x7f176c15afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33670/0 (socket says 172.21.15.81:33670) 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 -- 172.21.15.81:0/311423590 learned_addr learned my addr 172.21.15.81:0/311423590 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1772000640 1 --2- 172.21.15.81:0/311423590 >> v2:172.21.15.106:3300/0 conn(0x7f176c0c8410 0x7f176c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1770c00640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c15daf0 0x7f176c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 -- 172.21.15.81:0/311423590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c15daf0 msgr2=0x7f176c0c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c15daf0 0x7f176c0c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 -- 172.21.15.81:0/311423590 >> v2:172.21.15.106:3300/0 conn(0x7f176c0c8410 msgr2=0x7f176c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 --2- 172.21.15.81:0/311423590 >> v2:172.21.15.106:3300/0 conn(0x7f176c0c8410 0x7f176c06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.362+0000 7f1771600640 1 -- 172.21.15.81:0/311423590 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c0cad30 con 0x7f176c154970 2024-05-27T06:53:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1771600640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c154970 0x7f176c15afe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f176c06fa60 tx=0x7f176001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfd817404878099 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f176be00640 1 -- 172.21.15.81:0/311423590 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17600b0070 con 0x7f176c154970 2024-05-27T06:53:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f176be00640 1 -- 172.21.15.81:0/311423590 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17600ab600 con 0x7f176c154970 2024-05-27T06:53:46.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f176be00640 1 -- 172.21.15.81:0/311423590 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17600ab9d0 con 0x7f176c154970 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 -- 172.21.15.81:0/311423590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c154970 msgr2=0x7f176c15afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c154970 0x7f176c15afe0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f176c06fa60 tx=0x7f176001c260 comp rx=0 tx=0).stop 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 -- 172.21.15.81:0/311423590 shutdown_connections 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 --2- 172.21.15.81:0/311423590 >> v2:172.21.15.106:3300/0 conn(0x7f176c0c8410 0x7f176c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c15daf0 0x7f176c0c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 --2- 172.21.15.81:0/311423590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c154970 0x7f176c15afe0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 -- 172.21.15.81:0/311423590 >> 172.21.15.81:0/311423590 conn(0x7f176c0cb2e0 msgr2=0x7f176c0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 -- 172.21.15.81:0/311423590 shutdown_connections 2024-05-27T06:53:46.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.366+0000 7f1774000640 1 -- 172.21.15.81:0/311423590 wait complete. 2024-05-27T06:53:46.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 Processor -- start 2024-05-27T06:53:46.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- start start 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c0c8410 0x7f176c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f176c154970 0x7f176c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f176c0c9520 con 0x7f176c15daf0 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f176c0c93a0 con 0x7f176c154970 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f176c15df00 con 0x7f176c0c8410 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33672/0 (socket says 172.21.15.81:33672) 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 -- 172.21.15.81:0/3969163829 learned_addr learned my addr 172.21.15.81:0/3969163829 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1771600640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c0c8410 0x7f176c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1770c00640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f176c154970 0x7f176c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 -- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c0c8410 msgr2=0x7f176c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c0c8410 0x7f176c0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 -- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f176c154970 msgr2=0x7f176c1f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f176c154970 0x7f176c1f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 -- 172.21.15.81:0/3969163829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c1f6850 con 0x7f176c15daf0 2024-05-27T06:53:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1772000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1764009fd0 tx=0x7f176400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764014420 con 0x7f176c15daf0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764019070 con 0x7f176c15daf0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764014800 con 0x7f176c15daf0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c0c5e60 con 0x7f176c15daf0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.370+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c0c6100 con 0x7f176c15daf0 2024-05-27T06:53:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f176c06a560 con 0x7f176c15daf0 2024-05-27T06:53:46.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f17640149a0 con 0x7f176c15daf0 2024-05-27T06:53:46.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1769600640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 0x7f174c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:46.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f176408fbc0 con 0x7f176c15daf0 2024-05-27T06:53:46.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1771600640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 0x7f174c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:46.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.374+0000 7f1771600640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 0x7f174c071910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f17600ad000 tx=0x7f1760008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:46.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.382+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f176405c310 con 0x7f176c15daf0 2024-05-27T06:53:46.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.670+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f176c06fa90 con 0x7f176c15daf0 2024-05-27T06:53:46.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1769600640 1 -- 172.21.15.81:0/3969163829 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f1764060610 con 0x7f176c15daf0 2024-05-27T06:53:46.665 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:46.665 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:46.665 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:46.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 msgr2=0x7f174c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:46.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 0x7f174c071910 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f17600ad000 tx=0x7f1760008040 comp rx=0 tx=0).stop 2024-05-27T06:53:46.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 msgr2=0x7f176c1490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:46.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1764009fd0 tx=0x7f176400e4b0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 shutdown_connections 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f174c06f400 0x7f174c071910 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f176c15daf0 0x7f176c1490a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f176c154970 0x7f176c1f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 --2- 172.21.15.81:0/3969163829 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f176c0c8410 0x7f176c0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 >> 172.21.15.81:0/3969163829 conn(0x7f176c0cb2e0 msgr2=0x7f176c073fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 shutdown_connections 2024-05-27T06:53:46.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:46.674+0000 7f1774000640 1 -- 172.21.15.81:0/3969163829 wait complete. 2024-05-27T06:53:46.690 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:48.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.878+0000 7fb614600640 1 Processor -- start 2024-05-27T06:53:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.878+0000 7fb614600640 1 -- start start 2024-05-27T06:53:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb614600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb60c153870 0x7fb60c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb614600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60c159440 con 0x7fb60c0cb5d0 2024-05-27T06:53:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb614600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60c0739c0 con 0x7fb60c0743b0 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb614600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb60c159700 con 0x7fb60c153870 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb60c153870 0x7fb60c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb60c153870 0x7fb60c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33674/0 (socket says 172.21.15.81:33674) 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 -- 172.21.15.81:0/3886604086 learned_addr learned my addr 172.21.15.81:0/3886604086 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611200640 1 --1- 172.21.15.81:0/3886604086 >> v1:172.21.15.81:6789/0 conn(0x7fb60c0cb5d0 0x7fb60c073ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34258/0 (socket says 172.21.15.81:34258) 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 -- 172.21.15.81:0/3886604086 >> v1:172.21.15.106:6789/0 conn(0x7fb60c0743b0 legacy=0x7fb60c163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 -- 172.21.15.81:0/3886604086 >> v1:172.21.15.81:6789/0 conn(0x7fb60c0cb5d0 legacy=0x7fb60c073ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:48.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.882+0000 7fb611c00640 1 -- 172.21.15.81:0/3886604086 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c1599e0 con 0x7fb60c153870 2024-05-27T06:53:48.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb611c00640 1 --2- 172.21.15.81:0/3886604086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c153870 0x7fb60c0c6080 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb600009870 tx=0x7fb60001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4ed1c15822bf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb60be00640 1 -- 172.21.15.81:0/3886604086 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6000b5070 con 0x7fb60c153870 2024-05-27T06:53:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb60be00640 1 -- 172.21.15.81:0/3886604086 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6000a8a50 con 0x7fb60c153870 2024-05-27T06:53:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb60be00640 1 -- 172.21.15.81:0/3886604086 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6000a8e20 con 0x7fb60c153870 2024-05-27T06:53:48.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 -- 172.21.15.81:0/3886604086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c153870 msgr2=0x7fb60c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:48.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 --2- 172.21.15.81:0/3886604086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c153870 0x7fb60c0c6080 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb600009870 tx=0x7fb60001ed50 comp rx=0 tx=0).stop 2024-05-27T06:53:48.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 -- 172.21.15.81:0/3886604086 shutdown_connections 2024-05-27T06:53:48.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 --2- 172.21.15.81:0/3886604086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c153870 0x7fb60c0c6080 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:48.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 -- 172.21.15.81:0/3886604086 >> 172.21.15.81:0/3886604086 conn(0x7fb60c06dca0 msgr2=0x7fb60c06e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:48.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 -- 172.21.15.81:0/3886604086 shutdown_connections 2024-05-27T06:53:48.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.886+0000 7fb614600640 1 -- 172.21.15.81:0/3886604086 wait complete. 2024-05-27T06:53:48.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 Processor -- start 2024-05-27T06:53:48.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 -- start start 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c0cb5d0 0x7fb60c0bb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb60c153870 0x7fb60c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c0cb260 con 0x7fb60c0743b0 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c159fa0 con 0x7fb60c0cb5d0 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb614600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb60c0cb3e0 con 0x7fb60c153870 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:48.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33676/0 (socket says 172.21.15.81:33676) 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 -- 172.21.15.81:0/671051706 learned_addr learned my addr 172.21.15.81:0/671051706 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611200640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c0cb5d0 0x7fb60c0bb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb612600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb60c153870 0x7fb60c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 -- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb60c153870 msgr2=0x7fb60c0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb60c153870 0x7fb60c0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 -- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c0cb5d0 msgr2=0x7fb60c0bb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c0cb5d0 0x7fb60c0bb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.890+0000 7fb611c00640 1 -- 172.21.15.81:0/671051706 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c156050 con 0x7fb60c0743b0 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb611c00640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb6000b3040 tx=0x7fb60001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:48.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6000c0070 con 0x7fb60c0743b0 2024-05-27T06:53:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6000ae790 con 0x7fb60c0743b0 2024-05-27T06:53:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c1ef9d0 con 0x7fb60c0743b0 2024-05-27T06:53:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6000b5070 con 0x7fb60c0743b0 2024-05-27T06:53:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c14bab0 con 0x7fb60c0743b0 2024-05-27T06:53:48.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb60c06a560 con 0x7fb60c0743b0 2024-05-27T06:53:48.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.894+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb6000c3070 con 0x7fb60c0743b0 2024-05-27T06:53:48.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.898+0000 7fb609600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 0x7fb5ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:48.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.898+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7fb60012cff0 con 0x7fb60c0743b0 2024-05-27T06:53:48.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.898+0000 7fb611200640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 0x7fb5ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:48.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.898+0000 7fb611200640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 0x7fb5ec071910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000f70 tx=0x7fb5f8007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:48.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:48.902+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb6000188b0 con 0x7fb60c0743b0 2024-05-27T06:53:49.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.142+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb60c0c7440 con 0x7fb60c0743b0 2024-05-27T06:53:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.142+0000 7fb609600640 1 -- 172.21.15.81:0/671051706 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7fb6000f96d0 con 0x7fb60c0743b0 2024-05-27T06:53:49.135 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:49.135 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 msgr2=0x7fb5ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 0x7fb5ec071910 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5f8000f70 tx=0x7fb5f8007e30 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 msgr2=0x7fb60c155b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb6000b3040 tx=0x7fb60001baf0 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 shutdown_connections 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb60c153870 0x7fb60c0c57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb5ec06f400 0x7fb5ec071910 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c0cb5d0 0x7fb60c0bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 --2- 172.21.15.81:0/671051706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb60c0743b0 0x7fb60c155b10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 >> 172.21.15.81:0/671051706 conn(0x7fb60c06dca0 msgr2=0x7fb60c151de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 shutdown_connections 2024-05-27T06:53:49.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:49.146+0000 7fb614600640 1 -- 172.21.15.81:0/671051706 wait complete. 2024-05-27T06:53:49.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:51.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:51.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.342+0000 7f62a2e00640 1 Processor -- start 2024-05-27T06:53:51.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.342+0000 7f62a2e00640 1 -- start start 2024-05-27T06:53:51.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a2e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 0x7f629c156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:51.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a2e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629c15cc00 con 0x7f629c074c30 2024-05-27T06:53:51.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a2e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629c15cec0 con 0x7f629c15c6c0 2024-05-27T06:53:51.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a2e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f629c0744c0 con 0x7f629c0c8620 2024-05-27T06:53:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a0e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f629c074c30 0x7f629c075040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34262/0 (socket says 172.21.15.81:34262) 2024-05-27T06:53:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f62a0e00640 1 -- 172.21.15.81:0/2659487570 learned_addr learned my addr 172.21.15.81:0/2659487570 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629b400640 1 --2- 172.21.15.81:0/2659487570 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 0x7f629c156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769037231 0 0) 0x7f629c15cc00 con 0x7f629c074c30 2024-05-27T06:53:51.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6280003530 con 0x7f629c074c30 2024-05-27T06:53:51.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2381977504 0 0) 0x7f629c15cec0 con 0x7f629c15c6c0 2024-05-27T06:53:51.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f629c15cc00 con 0x7f629c15c6c0 2024-05-27T06:53:51.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629b400640 1 -- 172.21.15.81:0/2659487570 >> v1:172.21.15.106:6789/0 conn(0x7f629c15c6c0 legacy=0x7f629c073740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629b400640 1 -- 172.21.15.81:0/2659487570 >> v1:172.21.15.81:6789/0 conn(0x7f629c074c30 legacy=0x7f629c075040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.346+0000 7f629b400640 1 -- 172.21.15.81:0/2659487570 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c15d1a0 con 0x7f629c0c8620 2024-05-27T06:53:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.350+0000 7f629b400640 1 --2- 172.21.15.81:0/2659487570 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 0x7f629c156ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62880097f0 tx=0x7f628801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2cdf834959067de server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.350+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880b1070 con 0x7f629c0c8620 2024-05-27T06:53:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.350+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62880a8050 con 0x7f629c0c8620 2024-05-27T06:53:51.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.350+0000 7f629aa00640 1 -- 172.21.15.81:0/2659487570 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880ac860 con 0x7f629c0c8620 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- 172.21.15.81:0/2659487570 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 msgr2=0x7f629c156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 --2- 172.21.15.81:0/2659487570 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 0x7f629c156ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f62880097f0 tx=0x7f628801c420 comp rx=0 tx=0).stop 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- 172.21.15.81:0/2659487570 shutdown_connections 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 --2- 172.21.15.81:0/2659487570 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c0c8620 0x7f629c156ad0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- 172.21.15.81:0/2659487570 >> 172.21.15.81:0/2659487570 conn(0x7f629c071770 msgr2=0x7f629c0ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- 172.21.15.81:0/2659487570 shutdown_connections 2024-05-27T06:53:51.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- 172.21.15.81:0/2659487570 wait complete. 2024-05-27T06:53:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 Processor -- start 2024-05-27T06:53:51.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.354+0000 7f62a2e00640 1 -- start start 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c074c30 0x7f629c1f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c15c6c0 0x7f629c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f629c073fd0 con 0x7f629c0c8620 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f629c073e50 con 0x7f629c074c30 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f629c16a7b0 con 0x7f629c15c6c0 2024-05-27T06:53:51.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:51.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33692/0 (socket says 172.21.15.81:33692) 2024-05-27T06:53:51.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 -- 172.21.15.81:0/3489792949 learned_addr learned my addr 172.21.15.81:0/3489792949 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:51.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a0e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c15c6c0 0x7f629c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:51.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629be00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c074c30 0x7f629c1f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 -- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c15c6c0 msgr2=0x7f629c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c15c6c0 0x7f629c14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 -- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c074c30 msgr2=0x7f629c1f93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c074c30 0x7f629c1f93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 -- 172.21.15.81:0/3489792949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c1f9930 con 0x7f629c0c8620 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f629b400640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f62880097f0 tx=0x7f62880191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:51.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880b1040 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62880bb070 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629c0c5630 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62880b5410 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.358+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629c0c76a0 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.362+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629c06a560 con 0x7f629c0c8620 2024-05-27T06:53:51.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.362+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f62880b55b0 con 0x7f629c0c8620 2024-05-27T06:53:51.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.362+0000 7f6293e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 0x7f627c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:51.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.362+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f6288128de0 con 0x7f629c0c8620 2024-05-27T06:53:51.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.362+0000 7f629be00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 0x7f627c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:51.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.366+0000 7f629be00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 0x7f627c071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f628c007fd0 tx=0x7f628c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:51.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.370+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f62880160d0 con 0x7f629c0c8620 2024-05-27T06:53:51.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.622+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f629c06ccc0 con 0x7f629c0c8620 2024-05-27T06:53:51.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.622+0000 7f6293e00640 1 -- 172.21.15.81:0/3489792949 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f62880f54a0 con 0x7f629c0c8620 2024-05-27T06:53:51.615 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:51.615 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:51.615 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 msgr2=0x7f627c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 0x7f627c071910 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f628c007fd0 tx=0x7f628c009040 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 msgr2=0x7f629c152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f62880097f0 tx=0x7f62880191c0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 shutdown_connections 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f629c15c6c0 0x7f629c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f627c06f400 0x7f627c071910 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f629c0c8620 0x7f629c152700 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 --2- 172.21.15.81:0/3489792949 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c074c30 0x7f629c1f93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:51.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 >> 172.21.15.81:0/3489792949 conn(0x7f629c071770 msgr2=0x7f629c154a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:51.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 shutdown_connections 2024-05-27T06:53:51.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:51.626+0000 7f62a2e00640 1 -- 172.21.15.81:0/3489792949 wait complete. 2024-05-27T06:53:51.640 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:53.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:53.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.822+0000 7f7cac400640 1 Processor -- start 2024-05-27T06:53:53.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.822+0000 7f7cac400640 1 -- start start 2024-05-27T06:53:53.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7cac400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4154970 0x7f7ca415afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7cac400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ca4162e70 0x7f7ca4165290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7cac400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ca406f0f0 con 0x7f7ca415daf0 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7cac400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca406eec0 con 0x7f7ca4162e70 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7cac400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca406f3b0 con 0x7f7ca4154970 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ca4162e70 0x7f7ca4165290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ca4162e70 0x7f7ca4165290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33706/0 (socket says 172.21.15.81:33706) 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 -- 172.21.15.81:0/3756907538 learned_addr learned my addr 172.21.15.81:0/3756907538 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7ca9a00640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4154970 0x7f7ca415afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 -- 172.21.15.81:0/3756907538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4154970 msgr2=0x7f7ca415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:53.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4154970 0x7f7ca415afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:53.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 -- 172.21.15.81:0/3756907538 >> v1:172.21.15.81:6789/0 conn(0x7f7ca415daf0 legacy=0x7f7ca40c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:53.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.826+0000 7f7caa400640 1 -- 172.21.15.81:0/3756907538 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca40cad30 con 0x7f7ca4162e70 2024-05-27T06:53:53.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7caa400640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4162e70 0x7f7ca4165290 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009e50 tx=0x7f7c9c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205d6affa8c250da server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:53.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7ca3e00640 1 -- 172.21.15.81:0/3756907538 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c0a5700 con 0x7f7ca4162e70 2024-05-27T06:53:53.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7ca3e00640 1 -- 172.21.15.81:0/3756907538 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c9c0a58a0 con 0x7f7ca4162e70 2024-05-27T06:53:53.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7ca3e00640 1 -- 172.21.15.81:0/3756907538 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c0a5c70 con 0x7f7ca4162e70 2024-05-27T06:53:53.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 -- 172.21.15.81:0/3756907538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4162e70 msgr2=0x7f7ca4165290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:53.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4162e70 0x7f7ca4165290 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009e50 tx=0x7f7c9c01ca30 comp rx=0 tx=0).stop 2024-05-27T06:53:53.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 -- 172.21.15.81:0/3756907538 shutdown_connections 2024-05-27T06:53:53.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4162e70 0x7f7ca4165290 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:53.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 --2- 172.21.15.81:0/3756907538 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4154970 0x7f7ca415afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:53.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 -- 172.21.15.81:0/3756907538 >> 172.21.15.81:0/3756907538 conn(0x7f7ca40cb2e0 msgr2=0x7f7ca40cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:53.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 -- 172.21.15.81:0/3756907538 shutdown_connections 2024-05-27T06:53:53.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.830+0000 7f7cac400640 1 -- 172.21.15.81:0/3756907538 wait complete. 2024-05-27T06:53:53.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.834+0000 7f7cac400640 1 Processor -- start 2024-05-27T06:53:53.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.834+0000 7f7cac400640 1 -- start start 2024-05-27T06:53:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 0x7f7ca41579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4162e70 0x7f7ca4164ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca40c9520 con 0x7f7ca4154970 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca40c93a0 con 0x7f7ca415daf0 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca415df00 con 0x7f7ca4162e70 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33712/0 (socket says 172.21.15.81:33712) 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 -- 172.21.15.81:0/2472582225 learned_addr learned my addr 172.21.15.81:0/2472582225 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9000640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 0x7f7ca41579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7caa400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4162e70 0x7f7ca4164ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 -- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4162e70 msgr2=0x7f7ca4164ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4162e70 0x7f7ca4164ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 -- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 msgr2=0x7f7ca41579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 0x7f7ca41579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:53.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 -- 172.21.15.81:0/2472582225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca41f32f0 con 0x7f7ca4154970 2024-05-27T06:53:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9000640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 0x7f7ca41579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca9a00640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800e840 tx=0x7f7c9800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:53.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98017070 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c9800ee90 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca4155210 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9800cb60 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca40c8f80 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.838+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca406a560 con 0x7f7ca4154970 2024-05-27T06:53:53.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.842+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7c9801a070 con 0x7f7ca4154970 2024-05-27T06:53:53.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.842+0000 7f7ca1600640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 0x7f7c84071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:53.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.842+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f7c9808fad0 con 0x7f7ca4154970 2024-05-27T06:53:53.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.842+0000 7f7ca9000640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 0x7f7c84071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:53.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.842+0000 7f7ca9000640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 0x7f7c84071910 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ca41f7710 tx=0x7f7c90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:53.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:53.846+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7c9805c8e0 con 0x7f7ca4154970 2024-05-27T06:53:54.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.090+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ca406fa90 con 0x7f7ca4154970 2024-05-27T06:53:54.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.090+0000 7f7ca1600640 1 -- 172.21.15.81:0/2472582225 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f7c9805c1e0 con 0x7f7ca4154970 2024-05-27T06:53:54.084 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:54.084 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:54.084 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 msgr2=0x7f7c84071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 0x7f7c84071910 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ca41f7710 tx=0x7f7c90009040 comp rx=0 tx=0).stop 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 msgr2=0x7f7ca4155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800e840 tx=0x7f7c9800ec40 comp rx=0 tx=0).stop 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 shutdown_connections 2024-05-27T06:53:54.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7ca4162e70 0x7f7ca4164ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7c8406f400 0x7f7c84071910 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ca415daf0 0x7f7ca41579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 --2- 172.21.15.81:0/2472582225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca4154970 0x7f7ca4155dd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 >> 172.21.15.81:0/2472582225 conn(0x7f7ca40cb2e0 msgr2=0x7f7ca40ccc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 shutdown_connections 2024-05-27T06:53:54.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:54.094+0000 7f7cac400640 1 -- 172.21.15.81:0/2472582225 wait complete. 2024-05-27T06:53:54.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:56.109 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:56.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.278+0000 7f6998e00640 1 Processor -- start 2024-05-27T06:53:56.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.278+0000 7f6998e00640 1 -- start start 2024-05-27T06:53:56.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6998e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6994154880 0x7f6994070530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6998e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69940c8f30 con 0x7f69940732c0 2024-05-27T06:53:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6998e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6994075020 con 0x7f69940c8880 2024-05-27T06:53:56.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6998e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f69940755e0 con 0x7f6994154880 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f69940732c0 0x7f6994164c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40034/0 (socket says 172.21.15.81:40034) 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993e00640 1 -- 172.21.15.81:0/3443964686 learned_addr learned my addr 172.21.15.81:0/3443964686 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993400640 1 --2- 172.21.15.81:0/3443964686 >> v2:172.21.15.106:3300/0 conn(0x7f6994154880 0x7f6994070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3171516731 0 0) 0x7f69940c8f30 con 0x7f69940732c0 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6978003530 con 0x7f69940732c0 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993400640 1 -- 172.21.15.81:0/3443964686 >> v1:172.21.15.106:6789/0 conn(0x7f69940c8880 legacy=0x7f6994072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841634646 0 0) 0x7f6994075020 con 0x7f69940c8880 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993400640 1 -- 172.21.15.81:0/3443964686 >> v1:172.21.15.81:6789/0 conn(0x7f69940732c0 legacy=0x7f6994164c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.282+0000 7f6993400640 1 -- 172.21.15.81:0/3443964686 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6994075300 con 0x7f6994154880 2024-05-27T06:53:56.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6993400640 1 --2- 172.21.15.81:0/3443964686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f6994070530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69840097c0 tx=0x7f698401ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d481171d7c2c0a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:56.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69840b5070 con 0x7f6994154880 2024-05-27T06:53:56.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69840a8590 con 0x7f6994154880 2024-05-27T06:53:56.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6992000640 1 -- 172.21.15.81:0/3443964686 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c001aa0 con 0x7f6994154880 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 -- 172.21.15.81:0/3443964686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 msgr2=0x7f6994070530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 --2- 172.21.15.81:0/3443964686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f6994070530 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69840097c0 tx=0x7f698401ead0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 -- 172.21.15.81:0/3443964686 shutdown_connections 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 --2- 172.21.15.81:0/3443964686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f6994070530 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 -- 172.21.15.81:0/3443964686 >> 172.21.15.81:0/3443964686 conn(0x7f699406d2d0 msgr2=0x7f69940cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 -- 172.21.15.81:0/3443964686 shutdown_connections 2024-05-27T06:53:56.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.286+0000 7f6998e00640 1 -- 172.21.15.81:0/3443964686 wait complete. 2024-05-27T06:53:56.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 Processor -- start 2024-05-27T06:53:56.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 -- start start 2024-05-27T06:53:56.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940c8880 0x7f69940c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f699413e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f69940686b0 con 0x7f69940732c0 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f69940ca2d0 con 0x7f6994154880 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6998e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f69941f21a0 con 0x7f69940c8880 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6993400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6993400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35196/0 (socket says 172.21.15.81:35196) 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6993400640 1 -- 172.21.15.81:0/3992744640 learned_addr learned my addr 172.21.15.81:0/3992744640 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.290+0000 7f6993e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f699413e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6992a00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940c8880 0x7f69940c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 -- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940c8880 msgr2=0x7f69940c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940c8880 0x7f69940c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 -- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 msgr2=0x7f699413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f699413e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 -- 172.21.15.81:0/3992744640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6994151b10 con 0x7f69940732c0 2024-05-27T06:53:56.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f699413e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:56.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6993400640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69840b3040 tx=0x7f698401b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69840b5040 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69840bf070 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69940c5560 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69840a9bd0 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69940c5840 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6994050870 con 0x7f69940732c0 2024-05-27T06:53:56.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.294+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6984003620 con 0x7f69940732c0 2024-05-27T06:53:56.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.298+0000 7f698b400640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 0x7f6974071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:56.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.298+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6169+0+0 (secure 0 0 0) 0x7f698412c410 con 0x7f69940732c0 2024-05-27T06:53:56.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.298+0000 7f6992a00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 0x7f6974071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:56.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.298+0000 7f6992a00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 0x7f6974071940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f69940cd050 tx=0x7f6980009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:56.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.302+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f69840c9020 con 0x7f69940732c0 2024-05-27T06:53:56.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69940c7ce0 con 0x7f69940732c0 2024-05-27T06:53:56.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f698b400640 1 -- 172.21.15.81:0/3992744640 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6911 (secure 0 0 0) 0x7f69840186f0 con 0x7f69940732c0 2024-05-27T06:53:56.570 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:56.571 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:25.675481+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4287,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":6,"addr":"172.21.15.81:6839/328699254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":328699254},{"type":"v1","addr":"172.21.15.81:6839","nonce":328699254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5049,4870,4302,5184,4287]},"id":1}]} 2024-05-27T06:53:56.571 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 41 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 msgr2=0x7f6974071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 0x7f6974071940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f69940cd050 tx=0x7f6980009040 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 msgr2=0x7f69941562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.578+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f69840b3040 tx=0x7f698401b870 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 shutdown_connections 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69940c8880 0x7f69940c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f430 0x7f6974071940 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6994154880 0x7f699413e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 --2- 172.21.15.81:0/3992744640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69940732c0 0x7f69941562f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 >> 172.21.15.81:0/3992744640 conn(0x7f699406d2d0 msgr2=0x7f699414ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:56.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 shutdown_connections 2024-05-27T06:53:56.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:56.582+0000 7f6998e00640 1 -- 172.21.15.81:0/3992744640 wait complete. 2024-05-27T06:53:56.593 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-05-27T06:53:58.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.786+0000 7f727f200640 1 Processor -- start 2024-05-27T06:53:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.786+0000 7f727f200640 1 -- start start 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7280154780 0x7f7280072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280072a60 0x7f7280164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727f200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72800c9c70 con 0x7f72800c9630 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f72800c9a40 con 0x7f7280154780 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f728015b0c0 con 0x7f7280072a60 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7280154780 0x7f7280072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7280154780 0x7f7280072520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35212/0 (socket says 172.21.15.81:35212) 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 -- 172.21.15.81:0/657850099 learned_addr learned my addr 172.21.15.81:0/657850099 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f727e800640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280072a60 0x7f7280164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 -- 172.21.15.81:0/657850099 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280072a60 msgr2=0x7f7280164e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280072a60 0x7f7280164e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 -- 172.21.15.81:0/657850099 >> v1:172.21.15.106:6789/0 conn(0x7f72800c9630 legacy=0x7f728015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:58.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.790+0000 7f7277e00640 1 -- 172.21.15.81:0/657850099 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728015a4d0 con 0x7f7280154780 2024-05-27T06:53:58.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f7277e00640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7280154780 0x7f7280072520 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7268009870 tx=0x7f726801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5be233c3501a123 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:58.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727d400640 1 -- 172.21.15.81:0/657850099 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680ab610 con 0x7f7280154780 2024-05-27T06:53:58.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727d400640 1 -- 172.21.15.81:0/657850099 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72680b0070 con 0x7f7280154780 2024-05-27T06:53:58.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727d400640 1 -- 172.21.15.81:0/657850099 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680ab9d0 con 0x7f7280154780 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 -- 172.21.15.81:0/657850099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7280154780 msgr2=0x7f7280072520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7280154780 0x7f7280072520 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7268009870 tx=0x7f726801c500 comp rx=0 tx=0).stop 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 -- 172.21.15.81:0/657850099 shutdown_connections 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280072a60 0x7f7280164e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 --2- 172.21.15.81:0/657850099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7280154780 0x7f7280072520 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 -- 172.21.15.81:0/657850099 >> 172.21.15.81:0/657850099 conn(0x7f7280068880 msgr2=0x7f7280068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 -- 172.21.15.81:0/657850099 shutdown_connections 2024-05-27T06:53:58.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.794+0000 7f727f200640 1 -- 172.21.15.81:0/657850099 wait complete. 2024-05-27T06:53:58.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 Processor -- start 2024-05-27T06:53:58.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 -- start start 2024-05-27T06:53:58.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7280072a60 0x7f72801f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280154780 0x7f7280159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72800cd280 con 0x7f72800c9630 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f728015ad50 con 0x7f7280072a60 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727f200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f728015e1d0 con 0x7f7280154780 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35222/0 (socket says 172.21.15.81:35222) 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 -- 172.21.15.81:0/1501672958 learned_addr learned my addr 172.21.15.81:0/1501672958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727de00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7280072a60 0x7f72801f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f727e800640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280154780 0x7f7280159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 -- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280154780 msgr2=0x7f7280159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:58.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280154780 0x7f7280159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:58.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 -- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7280072a60 msgr2=0x7f72801f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:58.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7280072a60 0x7f72801f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.798+0000 7f7277e00640 1 -- 172.21.15.81:0/1501672958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72801f5f90 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f7277e00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7268019170 tx=0x7f726801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680be5f0 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72680be790 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72801513f0 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72680beb80 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72801f8fc0 con 0x7f72800c9630 2024-05-27T06:53:58.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7280050870 con 0x7f72800c9630 2024-05-27T06:53:58.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.802+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f72680b43f0 con 0x7f72800c9630 2024-05-27T06:53:58.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.806+0000 7f7276000640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 0x7f725c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:58.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.806+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f7268129b40 con 0x7f72800c9630 2024-05-27T06:53:58.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.806+0000 7f727de00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 0x7f725c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:58.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.806+0000 7f727de00640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 0x7f725c071910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7270007fd0 tx=0x7f7270009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:58.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:58.810+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f72680c4060 con 0x7f72800c9630 2024-05-27T06:53:59.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7280153fd0 con 0x7f72800c9630 2024-05-27T06:53:59.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f7276000640 1 -- 172.21.15.81:0/1501672958 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6085 (secure 0 0 0) 0x7f72680a8050 con 0x7f72800c9630 2024-05-27T06:53:59.022 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:59.022 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:58.363860+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,5049,5184]},"id":1}]} 2024-05-27T06:53:59.022 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 msgr2=0x7f725c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 0x7f725c071910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7270007fd0 tx=0x7f7270009040 comp rx=0 tx=0).stop 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 msgr2=0x7f7280150eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7268019170 tx=0x7f726801ada0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 shutdown_connections 2024-05-27T06:53:59.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7280154780 0x7f7280159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f725c06f400 0x7f725c071910 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72800c9630 0x7f7280150eb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 --2- 172.21.15.81:0/1501672958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7280072a60 0x7f72801f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.030+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 >> 172.21.15.81:0/1501672958 conn(0x7f7280068880 msgr2=0x7f7280068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.034+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 shutdown_connections 2024-05-27T06:53:59.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.034+0000 7f727f200640 1 -- 172.21.15.81:0/1501672958 wait complete. 2024-05-27T06:53:59.046 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-05-27T06:53:59.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:53:59.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:53:59.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 Processor -- start 2024-05-27T06:53:59.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 -- start start 2024-05-27T06:53:59.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 0x7fdf6c153460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:59.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6c0ca0c0 con 0x7fdf6c0cbce0 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf6c159ee0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf71800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c159cb0 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf70e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdf6c0cbce0 0x7fdf6c163bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40044/0 (socket says 172.21.15.81:40044) 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf70e00640 1 -- 172.21.15.81:0/274721458 learned_addr learned my addr 172.21.15.81:0/274721458 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6be00640 1 --2- 172.21.15.81:0/274721458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 0x7fdf6c153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:59.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6aa00640 1 -- 172.21.15.81:0/274721458 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1008822900 0 0) 0x7fdf6c0ca0c0 con 0x7fdf6c0cbce0 2024-05-27T06:53:59.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6aa00640 1 -- 172.21.15.81:0/274721458 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf50003530 con 0x7fdf6c0cbce0 2024-05-27T06:53:59.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6be00640 1 -- 172.21.15.81:0/274721458 >> v1:172.21.15.106:6789/0 conn(0x7fdf6c0ceac0 legacy=0x7fdf6c0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6be00640 1 -- 172.21.15.81:0/274721458 >> v1:172.21.15.81:6789/0 conn(0x7fdf6c0cbce0 legacy=0x7fdf6c163bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.222+0000 7fdf6be00640 1 -- 172.21.15.81:0/274721458 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c0ca360 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf6be00640 1 --2- 172.21.15.81:0/274721458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 0x7fdf6c153460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009770 tx=0x7fdf5c01c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5b951a7bc81309 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:59.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf6aa00640 1 -- 172.21.15.81:0/274721458 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0b0070 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf6aa00640 1 -- 172.21.15.81:0/274721458 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0ab600 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf6aa00640 1 -- 172.21.15.81:0/274721458 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0ab9d0 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf71800640 1 -- 172.21.15.81:0/274721458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 msgr2=0x7fdf6c153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf71800640 1 --2- 172.21.15.81:0/274721458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 0x7fdf6c153460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009770 tx=0x7fdf5c01c310 comp rx=0 tx=0).stop 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf71800640 1 -- 172.21.15.81:0/274721458 shutdown_connections 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf71800640 1 --2- 172.21.15.81:0/274721458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cacf0 0x7fdf6c153460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.226+0000 7fdf71800640 1 -- 172.21.15.81:0/274721458 >> 172.21.15.81:0/274721458 conn(0x7fdf6c06da70 msgr2=0x7fdf6c06de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.230+0000 7fdf71800640 1 -- 172.21.15.81:0/274721458 shutdown_connections 2024-05-27T06:53:59.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.230+0000 7fdf71800640 1 -- 172.21.15.81:0/274721458 wait complete. 2024-05-27T06:53:59.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.230+0000 7fdf71800640 1 Processor -- start 2024-05-27T06:53:59.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.230+0000 7fdf71800640 1 -- start start 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf6c0cacf0 0x7fdf6c14f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 0x7fdf6c1f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c15a1c0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c0734b0 con 0x7fdf6c0cacf0 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c1539a0 con 0x7fdf6c0cbce0 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35230/0 (socket says 172.21.15.81:35230) 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 -- 172.21.15.81:0/1114894747 learned_addr learned my addr 172.21.15.81:0/1114894747 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf6be00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf6c0cacf0 0x7fdf6c14f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:59.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf6b400640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 0x7fdf6c1f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 -- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 msgr2=0x7fdf6c1f9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 0x7fdf6c1f9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 -- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf6c0cacf0 msgr2=0x7fdf6c14f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf6c0cacf0 0x7fdf6c14f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 -- 172.21.15.81:0/1114894747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c1f9590 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf6b400640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 0x7fdf6c1f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:53:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf70e00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400cad0 tx=0x7fdf6400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400ac30 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6400add0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c14ff70 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64012600 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c15a490 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.234+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c050870 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.238+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdf640127a0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.238+0000 7fdf63e00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 0x7fdf4c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:53:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.238+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fdf6408fa30 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.238+0000 7fdf6be00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 0x7fdf4c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:53:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.238+0000 7fdf6be00640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 0x7fdf4c071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0ad000 tx=0x7fdf5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:53:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.242+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdf6405c6f0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.490+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf6c1650b0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.490+0000 7fdf63e00640 1 -- 172.21.15.81:0/1114894747 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6085 (secure 0 0 0) 0x7fdf6405bff0 con 0x7fdf6c0ceac0 2024-05-27T06:53:59.482 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:53:59.483 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:58.363860+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,5049,5184]},"id":1}]} 2024-05-27T06:53:59.483 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 msgr2=0x7fdf4c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 0x7fdf4c071940 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0ad000 tx=0x7fdf5c008040 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 msgr2=0x7fdf6c14c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400cad0 tx=0x7fdf6400cf90 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 shutdown_connections 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdf6c0cbce0 0x7fdf6c1f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdf4c06f430 0x7fdf4c071940 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf6c0ceac0 0x7fdf6c14c730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 --2- 172.21.15.81:0/1114894747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf6c0cacf0 0x7fdf6c14f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 >> 172.21.15.81:0/1114894747 conn(0x7fdf6c06da70 msgr2=0x7fdf6c157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 shutdown_connections 2024-05-27T06:53:59.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:53:59.494+0000 7fdf71800640 1 -- 172.21.15.81:0/1114894747 wait complete. 2024-05-27T06:53:59.505 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:53:58.363860+0000", "qdb_cluster": [ 4302, 4870, 5049, 5184 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:54:01.506 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.686+0000 7f462ee00640 1 Processor -- start 2024-05-27T06:54:01.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.686+0000 7f462ee00640 1 -- start start 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ee00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 0x7f46280cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ee00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f462815e0f0 0x7f462815e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ee00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f462815ca00 con 0x7f4628156be0 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ee00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f462815dc60 con 0x7f462815e0f0 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ee00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f46280cda90 con 0x7f46280cd140 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f462815e0f0 0x7f462815e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f4627400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 0x7f46280cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f462815e0f0 0x7f462815e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35240/0 (socket says 172.21.15.81:35240) 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 -- 172.21.15.81:0/3701110958 learned_addr learned my addr 172.21.15.81:0/3701110958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:01.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f4627e00640 1 --1- 172.21.15.81:0/3701110958 >> v1:172.21.15.106:6789/0 conn(0x7f4628156be0 0x7f46280d1f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:33184/0 (socket says 172.21.15.81:33184) 2024-05-27T06:54:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f4626a00640 1 -- 172.21.15.81:0/3701110958 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704387285 0 0) 0x7f462815ca00 con 0x7f4628156be0 2024-05-27T06:54:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f4626a00640 1 -- 172.21.15.81:0/3701110958 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f460c003530 con 0x7f4628156be0 2024-05-27T06:54:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 -- 172.21.15.81:0/3701110958 >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 msgr2=0x7f46280cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 --2- 172.21.15.81:0/3701110958 >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 0x7f46280cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 -- 172.21.15.81:0/3701110958 >> v1:172.21.15.106:6789/0 conn(0x7f4628156be0 legacy=0x7f46280d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f462ce00640 1 -- 172.21.15.81:0/3701110958 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462815cce0 con 0x7f462815e0f0 2024-05-27T06:54:01.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.690+0000 7f4627400640 1 --2- 172.21.15.81:0/3701110958 >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 0x7f46280cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:54:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ce00640 1 --2- 172.21.15.81:0/3701110958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f462815e500 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f46280cf780 tx=0x7f462001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eebef6a2bb922d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f4626a00640 1 -- 172.21.15.81:0/3701110958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200b4070 con 0x7f462815e0f0 2024-05-27T06:54:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f4626a00640 1 -- 172.21.15.81:0/3701110958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46200af400 con 0x7f462815e0f0 2024-05-27T06:54:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f4626a00640 1 -- 172.21.15.81:0/3701110958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200af7d0 con 0x7f462815e0f0 2024-05-27T06:54:01.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 -- 172.21.15.81:0/3701110958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 msgr2=0x7f462815e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 --2- 172.21.15.81:0/3701110958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f462815e500 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f46280cf780 tx=0x7f462001f1e0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 -- 172.21.15.81:0/3701110958 shutdown_connections 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 --2- 172.21.15.81:0/3701110958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f462815e500 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 --2- 172.21.15.81:0/3701110958 >> v2:172.21.15.106:3300/0 conn(0x7f46280cd140 0x7f46280cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 -- 172.21.15.81:0/3701110958 >> 172.21.15.81:0/3701110958 conn(0x7f462806e8f0 msgr2=0x7f462806ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 -- 172.21.15.81:0/3701110958 shutdown_connections 2024-05-27T06:54:01.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.694+0000 7f462ee00640 1 -- 172.21.15.81:0/3701110958 wait complete. 2024-05-27T06:54:01.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 Processor -- start 2024-05-27T06:54:01.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 -- start start 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46280cd140 0x7f4628159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4628156be0 0x7f46280cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f462815d580 con 0x7f462815e0f0 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f462815d2a0 con 0x7f46280cd140 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ee00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f462815ea40 con 0x7f4628156be0 2024-05-27T06:54:01.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35242/0 (socket says 172.21.15.81:35242) 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 -- 172.21.15.81:0/683969504 learned_addr learned my addr 172.21.15.81:0/683969504 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f4627e00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46280cd140 0x7f4628159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f4627400640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4628156be0 0x7f46280cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 -- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4628156be0 msgr2=0x7f46280cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4628156be0 0x7f46280cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 -- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46280cd140 msgr2=0x7f4628159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46280cd140 0x7f4628159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 -- 172.21.15.81:0/683969504 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4628159890 con 0x7f462815e0f0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f462ce00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f46200b2040 tx=0x7f462001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.698+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200c0070 con 0x7f462815e0f0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.702+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46200afbb0 con 0x7f462815e0f0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.702+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200b4040 con 0x7f462815e0f0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.702+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46280c57b0 con 0x7f462815e0f0 2024-05-27T06:54:01.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.702+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46280c6ee0 con 0x7f462815e0f0 2024-05-27T06:54:01.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.702+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f462806a560 con 0x7f462815e0f0 2024-05-27T06:54:01.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.706+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f46200c3070 con 0x7f462815e0f0 2024-05-27T06:54:01.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.706+0000 7f461fe00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 0x7f4608071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:01.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.706+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f462012cca0 con 0x7f462815e0f0 2024-05-27T06:54:01.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.706+0000 7f4627e00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 0x7f4608071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:01.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.706+0000 7f4627e00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 0x7f4608071940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4618007f60 tx=0x7f4618007ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:01.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.710+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f46200ca080 con 0x7f462815e0f0 2024-05-27T06:54:01.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46280cdc80 con 0x7f462815e0f0 2024-05-27T06:54:01.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f461fe00640 1 -- 172.21.15.81:0/683969504 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6085 (secure 0 0 0) 0x7f4620018d40 con 0x7f462815e0f0 2024-05-27T06:54:01.937 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:01.937 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:53:58.363860+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,5049,5184]},"id":1}]} 2024-05-27T06:54:01.938 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 42 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 msgr2=0x7f4608071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 0x7f4608071940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4618007f60 tx=0x7f4618007ef0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 msgr2=0x7f4628155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f46200b2040 tx=0x7f462001bf80 comp rx=0 tx=0).stop 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 shutdown_connections 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4628156be0 0x7f46280cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f460806f430 0x7f4608071940 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f462815e0f0 0x7f4628155f50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 --2- 172.21.15.81:0/683969504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46280cd140 0x7f4628159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:01.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 >> 172.21.15.81:0/683969504 conn(0x7f462806e8f0 msgr2=0x7f4628073ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:01.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 shutdown_connections 2024-05-27T06:54:01.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:01.946+0000 7f462ee00640 1 -- 172.21.15.81:0/683969504 wait complete. 2024-05-27T06:54:01.960 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:03.961 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:04.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 Processor -- start 2024-05-27T06:54:04.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 -- start start 2024-05-27T06:54:04.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd61815b050 0x7fd618154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 0x7fd6180cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd61815dfa0 0x7fd61815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd6180c9170 con 0x7fd61815b050 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd618074bf0 con 0x7fd6180cebf0 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd617e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd618154a90 con 0x7fd61815dfa0 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd616a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd61815b050 0x7fd618154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd616a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd61815b050 0x7fd618154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35246/0 (socket says 172.21.15.81:35246) 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.146+0000 7fd616a00640 1 -- 172.21.15.81:0/2687925782 learned_addr learned my addr 172.21.15.81:0/2687925782 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616000640 1 --2- 172.21.15.81:0/2687925782 >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 0x7fd6180cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd617400640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd61815dfa0 0x7fd61815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 -- 172.21.15.81:0/2687925782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd61815dfa0 msgr2=0x7fd61815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd61815dfa0 0x7fd61815e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 -- 172.21.15.81:0/2687925782 >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 msgr2=0x7fd6180cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 --2- 172.21.15.81:0/2687925782 >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 0x7fd6180cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 -- 172.21.15.81:0/2687925782 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6180c93a0 con 0x7fd61815b050 2024-05-27T06:54:04.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616000640 1 --2- 172.21.15.81:0/2687925782 >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 0x7fd6180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:54:04.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd616a00640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815b050 0x7fd618154550 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd618073330 tx=0x7fd60801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1894cfd4c15e6d70 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:04.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd615600640 1 -- 172.21.15.81:0/2687925782 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6080b5070 con 0x7fd61815b050 2024-05-27T06:54:04.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd615600640 1 -- 172.21.15.81:0/2687925782 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6080a89d0 con 0x7fd61815b050 2024-05-27T06:54:04.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.150+0000 7fd615600640 1 -- 172.21.15.81:0/2687925782 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6080a8da0 con 0x7fd61815b050 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- 172.21.15.81:0/2687925782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815b050 msgr2=0x7fd618154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815b050 0x7fd618154550 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd618073330 tx=0x7fd60801ed10 comp rx=0 tx=0).stop 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- 172.21.15.81:0/2687925782 shutdown_connections 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd61815dfa0 0x7fd61815e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 --2- 172.21.15.81:0/2687925782 >> v2:172.21.15.106:3300/0 conn(0x7fd6180cebf0 0x7fd6180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 --2- 172.21.15.81:0/2687925782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815b050 0x7fd618154550 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- 172.21.15.81:0/2687925782 >> 172.21.15.81:0/2687925782 conn(0x7fd618071390 msgr2=0x7fd6180717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- 172.21.15.81:0/2687925782 shutdown_connections 2024-05-27T06:54:04.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- 172.21.15.81:0/2687925782 wait complete. 2024-05-27T06:54:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 Processor -- start 2024-05-27T06:54:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.154+0000 7fd617e00640 1 -- start start 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6180cebf0 0x7fd61815c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd61815b050 0x7fd618151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd618075100 con 0x7fd61815dfa0 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd618074e20 con 0x7fd61815b050 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd61815e8f0 con 0x7fd6180cebf0 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35250/0 (socket says 172.21.15.81:35250) 2024-05-27T06:54:04.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 -- 172.21.15.81:0/244846768 learned_addr learned my addr 172.21.15.81:0/244846768 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd616a00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6180cebf0 0x7fd61815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd616000640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd61815b050 0x7fd618151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 -- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6180cebf0 msgr2=0x7fd61815c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6180cebf0 0x7fd61815c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 -- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd61815b050 msgr2=0x7fd618151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd61815b050 0x7fd618151e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 -- 172.21.15.81:0/244846768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6181f6940 con 0x7fd61815dfa0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617400640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd610009fd0 tx=0x7fd61000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610014420 con 0x7fd61815dfa0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd610019070 con 0x7fd61815dfa0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610014800 con 0x7fd61815dfa0 2024-05-27T06:54:04.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6180ce3a0 con 0x7fd61815dfa0 2024-05-27T06:54:04.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6180cb100 con 0x7fd61815dfa0 2024-05-27T06:54:04.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.158+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd618050870 con 0x7fd61815dfa0 2024-05-27T06:54:04.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.162+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd6100149a0 con 0x7fd61815dfa0 2024-05-27T06:54:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.162+0000 7fd60f400640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 0x7fd5f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.162+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fd61008fc80 con 0x7fd61815dfa0 2024-05-27T06:54:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.162+0000 7fd616a00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 0x7fd5f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.162+0000 7fd616a00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 0x7fd5f8071910 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6080b2000 tx=0x7fd608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:04.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.170+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd6181f6940 con 0x7fd61815dfa0 2024-05-27T06:54:04.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:54:04.237 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:54:04.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.402+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6180cbc60 con 0x7fd61815dfa0 2024-05-27T06:54:04.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.402+0000 7fd60f400640 1 -- 172.21.15.81:0/244846768 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6086 (secure 0 0 0) 0x7fd61005c9d0 con 0x7fd61815dfa0 2024-05-27T06:54:04.394 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:04.394 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:03.378021+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:resolve","state_seq":8,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4870,4302]},"id":1}]} 2024-05-27T06:54:04.395 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 43 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 msgr2=0x7fd5f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 0x7fd5f8071910 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6080b2000 tx=0x7fd608009040 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 msgr2=0x7fd61815ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd610009fd0 tx=0x7fd61000e4b0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 shutdown_connections 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5f806f400 0x7fd5f8071910 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd61815dfa0 0x7fd61815ba30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd61815b050 0x7fd618151e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 --2- 172.21.15.81:0/244846768 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd6180cebf0 0x7fd61815c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:04.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 >> 172.21.15.81:0/244846768 conn(0x7fd618071390 msgr2=0x7fd61806e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:04.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 shutdown_connections 2024-05-27T06:54:04.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:04.406+0000 7fd617e00640 1 -- 172.21.15.81:0/244846768 wait complete. 2024-05-27T06:54:04.420 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:06.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:06.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.602+0000 7f52fd200640 1 Processor -- start 2024-05-27T06:54:06.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.602+0000 7f52fd200640 1 -- start start 2024-05-27T06:54:06.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52fd200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 0x7f52f80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:06.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52fd200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f8072840 con 0x7f52f8161cd0 2024-05-27T06:54:06.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52fd200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f80c5270 con 0x7f52f8072430 2024-05-27T06:54:06.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52fd200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f52f80c5040 con 0x7f52f80c5b60 2024-05-27T06:54:06.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f6800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f52f8161cd0 0x7f52f81640f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59172/0 (socket says 172.21.15.81:59172) 2024-05-27T06:54:06.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f6800640 1 -- 172.21.15.81:0/934937264 learned_addr learned my addr 172.21.15.81:0/934937264 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52efe00640 1 --2- 172.21.15.81:0/934937264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 0x7f52f80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174857533 0 0) 0x7f52f8072840 con 0x7f52f8161cd0 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52dc003530 con 0x7f52f8161cd0 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998948657 0 0) 0x7f52f80c5270 con 0x7f52f8072430 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f8072840 con 0x7f52f8072430 2024-05-27T06:54:06.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 107217016 0 0) 0x7f52dc003530 con 0x7f52f8161cd0 2024-05-27T06:54:06.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52efe00640 1 -- 172.21.15.81:0/934937264 >> v1:172.21.15.106:6789/0 conn(0x7f52f8072430 legacy=0x7f52f8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52efe00640 1 -- 172.21.15.81:0/934937264 >> v1:172.21.15.81:6789/0 conn(0x7f52f8161cd0 legacy=0x7f52f81640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.606+0000 7f52efe00640 1 -- 172.21.15.81:0/934937264 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f80c7810 con 0x7f52f80c5b60 2024-05-27T06:54:06.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52efe00640 1 --2- 172.21.15.81:0/934937264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 0x7f52f80c5f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52e00097f0 tx=0x7f52e001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce4a6843b5110d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:06.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e00b0070 con 0x7f52f80c5b60 2024-05-27T06:54:06.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e00ab400 con 0x7f52f80c5b60 2024-05-27T06:54:06.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52f5400640 1 -- 172.21.15.81:0/934937264 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0001a80 con 0x7f52f80c5b60 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 -- 172.21.15.81:0/934937264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 msgr2=0x7f52f80c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 --2- 172.21.15.81:0/934937264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 0x7f52f80c5f70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52e00097f0 tx=0x7f52e001c400 comp rx=0 tx=0).stop 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 -- 172.21.15.81:0/934937264 shutdown_connections 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 --2- 172.21.15.81:0/934937264 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f80c5b60 0x7f52f80c5f70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 -- 172.21.15.81:0/934937264 >> 172.21.15.81:0/934937264 conn(0x7f52f8070700 msgr2=0x7f52f80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 -- 172.21.15.81:0/934937264 shutdown_connections 2024-05-27T06:54:06.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.610+0000 7f52fd200640 1 -- 172.21.15.81:0/934937264 wait complete. 2024-05-27T06:54:06.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 Processor -- start 2024-05-27T06:54:06.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 -- start start 2024-05-27T06:54:06.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f8072430 0x7f52f8148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:06.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:06.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52f8161cd0 0x7f52f81520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52f80c7dd0 con 0x7f52f80c5b60 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52f80c75c0 con 0x7f52f8161cd0 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52fd200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f52f80c5610 con 0x7f52f8072430 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35024/0 (socket says 172.21.15.81:35024) 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 -- 172.21.15.81:0/2711805864 learned_addr learned my addr 172.21.15.81:0/2711805864 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52f5e00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f8072430 0x7f52f8148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52f6800640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52f8161cd0 0x7f52f81520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 -- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f8072430 msgr2=0x7f52f8148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f8072430 0x7f52f8148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 -- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52f8161cd0 msgr2=0x7f52f81520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52f8161cd0 0x7f52f81520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 -- 172.21.15.81:0/2711805864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f80cb840 con 0x7f52f80c5b60 2024-05-27T06:54:06.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.614+0000 7f52efe00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f52e00097f0 tx=0x7f52e00191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:06.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e00b0040 con 0x7f52f80c5b60 2024-05-27T06:54:06.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e00abc90 con 0x7f52f80c5b60 2024-05-27T06:54:06.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0004220 con 0x7f52f80c5b60 2024-05-27T06:54:06.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f8147b60 con 0x7f52f80c5b60 2024-05-27T06:54:06.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81655b0 con 0x7f52f80c5b60 2024-05-27T06:54:06.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f806a560 con 0x7f52f80c5b60 2024-05-27T06:54:06.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.618+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f52e00bd020 con 0x7f52f80c5b60 2024-05-27T06:54:06.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.622+0000 7f52ee000640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 0x7f52d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:06.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.622+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f52e01288c0 con 0x7f52f80c5b60 2024-05-27T06:54:06.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.622+0000 7f52f5e00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 0x7f52d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:06.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.622+0000 7f52f5e00640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 0x7f52d4071910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52e8007fd0 tx=0x7f52e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:06.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.626+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f52e0015ff0 con 0x7f52f80c5b60 2024-05-27T06:54:06.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.894+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52f80c45d0 con 0x7f52f80c5b60 2024-05-27T06:54:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.894+0000 7f52ee000640 1 -- 172.21.15.81:0/2711805864 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6086 (secure 0 0 0) 0x7f52e00f5620 con 0x7f52f80c5b60 2024-05-27T06:54:06.888 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:06.888 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:03.378021+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:resolve","state_seq":8,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4870,4302]},"id":1}]} 2024-05-27T06:54:06.888 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 43 2024-05-27T06:54:06.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 msgr2=0x7f52d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 0x7f52d4071910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f52e8007fd0 tx=0x7f52e8009040 comp rx=0 tx=0).stop 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 msgr2=0x7f52f8147620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f52e00097f0 tx=0x7f52e00191a0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 shutdown_connections 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f52d406f400 0x7f52d4071910 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52f8161cd0 0x7f52f81520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f80c5b60 0x7f52f8147620 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 --2- 172.21.15.81:0/2711805864 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f52f8072430 0x7f52f8148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 >> 172.21.15.81:0/2711805864 conn(0x7f52f8070700 msgr2=0x7f52f8075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 shutdown_connections 2024-05-27T06:54:06.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:06.898+0000 7f52fd200640 1 -- 172.21.15.81:0/2711805864 wait complete. 2024-05-27T06:54:06.914 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:08.915 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:09.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.094+0000 7f9469600640 1 Processor -- start 2024-05-27T06:54:09.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.094+0000 7f9469600640 1 -- start start 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9469600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f94640c9630 0x7f9464154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9469600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f946415dac0 0x7f946415ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9469600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94640c9c70 con 0x7f9464072520 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9469600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f94640c9a40 con 0x7f946415dac0 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9469600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f946415b0c0 con 0x7f94640c9630 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f946415dac0 0x7f946415ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f946415dac0 0x7f946415ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35030/0 (socket says 172.21.15.81:35030) 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 -- 172.21.15.81:0/548378946 learned_addr learned my addr 172.21.15.81:0/548378946 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463e00640 1 --2- 172.21.15.81:0/548378946 >> v2:172.21.15.106:3300/0 conn(0x7f94640c9630 0x7f9464154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 -- 172.21.15.81:0/548378946 >> v2:172.21.15.106:3300/0 conn(0x7f94640c9630 msgr2=0x7f9464154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 --2- 172.21.15.81:0/548378946 >> v2:172.21.15.106:3300/0 conn(0x7f94640c9630 0x7f9464154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 -- 172.21.15.81:0/548378946 >> v1:172.21.15.106:6789/0 conn(0x7f9464072520 legacy=0x7f94640cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.098+0000 7f9463400640 1 -- 172.21.15.81:0/548378946 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946415a4d0 con 0x7f946415dac0 2024-05-27T06:54:09.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9463400640 1 --2- 172.21.15.81:0/548378946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f946415ded0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94640cc4a0 tx=0x7f945001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab8fc06a2733bbf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:09.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9462a00640 1 -- 172.21.15.81:0/548378946 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94500b1070 con 0x7f946415dac0 2024-05-27T06:54:09.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9462a00640 1 -- 172.21.15.81:0/548378946 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94500ac600 con 0x7f946415dac0 2024-05-27T06:54:09.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9462a00640 1 -- 172.21.15.81:0/548378946 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94500ac9d0 con 0x7f946415dac0 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 -- 172.21.15.81:0/548378946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 msgr2=0x7f946415ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 --2- 172.21.15.81:0/548378946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f946415ded0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f94640cc4a0 tx=0x7f945001c490 comp rx=0 tx=0).stop 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 -- 172.21.15.81:0/548378946 shutdown_connections 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 --2- 172.21.15.81:0/548378946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f946415ded0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 --2- 172.21.15.81:0/548378946 >> v2:172.21.15.106:3300/0 conn(0x7f94640c9630 0x7f9464154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 -- 172.21.15.81:0/548378946 >> 172.21.15.81:0/548378946 conn(0x7f9464068880 msgr2=0x7f9464068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:09.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 -- 172.21.15.81:0/548378946 shutdown_connections 2024-05-27T06:54:09.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.102+0000 7f9469600640 1 -- 172.21.15.81:0/548378946 wait complete. 2024-05-27T06:54:09.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 Processor -- start 2024-05-27T06:54:09.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 -- start start 2024-05-27T06:54:09.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9464072520 0x7f946414c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94640c9630 0x7f94640ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9464072d40 con 0x7f946415dac0 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f94640cd280 con 0x7f9464072520 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9469600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f94640ca7e0 con 0x7f94640c9630 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35038/0 (socket says 172.21.15.81:35038) 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 -- 172.21.15.81:0/3481606812 learned_addr learned my addr 172.21.15.81:0/3481606812 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9463e00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9464072520 0x7f946414c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9463400640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94640c9630 0x7f94640ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 -- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94640c9630 msgr2=0x7f94640ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94640c9630 0x7f94640ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 -- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9464072520 msgr2=0x7f946414c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9464072520 0x7f946414c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.106+0000 7f9468c00640 1 -- 172.21.15.81:0/3481606812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946414c730 con 0x7f946415dac0 2024-05-27T06:54:09.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f9468c00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f945c00c760 tx=0x7f945c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c003ba0 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f945c003d60 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c00aa90 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946414b410 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946414bcd0 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946406f6a0 con 0x7f946415dac0 2024-05-27T06:54:09.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.110+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f945c00ac30 con 0x7f946415dac0 2024-05-27T06:54:09.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.114+0000 7f945be00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 0x7f9444071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:09.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.114+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f945c08f250 con 0x7f946415dac0 2024-05-27T06:54:09.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.114+0000 7f9463e00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 0x7f9444071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:09.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.114+0000 7f9463e00640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 0x7f9444071910 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f94640c6cb0 tx=0x7f9454009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:09.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.118+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f945c05b8c0 con 0x7f946415dac0 2024-05-27T06:54:09.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.342+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946414b930 con 0x7f946415dac0 2024-05-27T06:54:09.337 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:09.337 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:08.699129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,5049,5184]},"id":1}]} 2024-05-27T06:54:09.337 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 44 2024-05-27T06:54:09.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f945be00640 1 -- 172.21.15.81:0/3481606812 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6089 (secure 0 0 0) 0x7f945c05fbc0 con 0x7f946415dac0 2024-05-27T06:54:09.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 msgr2=0x7f9444071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 0x7f9444071910 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f94640c6cb0 tx=0x7f9454009190 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 msgr2=0x7f9464147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f945c00c760 tx=0x7f945c00cc20 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 shutdown_connections 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94640c9630 0x7f94640ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f944406f400 0x7f9444071910 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f946415dac0 0x7f9464147020 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 --2- 172.21.15.81:0/3481606812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9464072520 0x7f946414c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 >> 172.21.15.81:0/3481606812 conn(0x7f9464068880 msgr2=0x7f94641532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 shutdown_connections 2024-05-27T06:54:09.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:09.346+0000 7f9469600640 1 -- 172.21.15.81:0/3481606812 wait complete. 2024-05-27T06:54:09.656 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:11.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:11.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.830+0000 7f928dc00640 1 Processor -- start 2024-05-27T06:54:11.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.830+0000 7f928dc00640 1 -- start start 2024-05-27T06:54:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f928dc00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f92880736c0 0x7f9288073c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f928dc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928806dd50 con 0x7f928806d120 2024-05-27T06:54:11.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f928dc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f928815d710 con 0x7f9288156c40 2024-05-27T06:54:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f928dc00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9288074770 con 0x7f92880736c0 2024-05-27T06:54:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9287400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f928806d120 0x7f9288167040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59184/0 (socket says 172.21.15.81:59184) 2024-05-27T06:54:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9287400640 1 -- 172.21.15.81:0/1162329727 learned_addr learned my addr 172.21.15.81:0/1162329727 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:54:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9286000640 1 --2- 172.21.15.81:0/1162329727 >> v2:172.21.15.106:3300/0 conn(0x7f92880736c0 0x7f9288073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:11.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9285600640 1 -- 172.21.15.81:0/1162329727 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30467225 0 0) 0x7f928806dd50 con 0x7f928806d120 2024-05-27T06:54:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9285600640 1 -- 172.21.15.81:0/1162329727 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f926c003530 con 0x7f928806d120 2024-05-27T06:54:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9286000640 1 -- 172.21.15.81:0/1162329727 >> v1:172.21.15.106:6789/0 conn(0x7f9288156c40 legacy=0x7f9288074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9286000640 1 -- 172.21.15.81:0/1162329727 >> v1:172.21.15.81:6789/0 conn(0x7f928806d120 legacy=0x7f9288167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:11.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.834+0000 7f9286000640 1 -- 172.21.15.81:0/1162329727 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928806c8c0 con 0x7f92880736c0 2024-05-27T06:54:11.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f9286000640 1 --2- 172.21.15.81:0/1162329727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288073c40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f92740097c0 tx=0x7f927401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e902b931bfa7abc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:11.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f9285600640 1 -- 172.21.15.81:0/1162329727 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92740b1070 con 0x7f92880736c0 2024-05-27T06:54:11.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f9285600640 1 -- 172.21.15.81:0/1162329727 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92740ac5f0 con 0x7f92880736c0 2024-05-27T06:54:11.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f9285600640 1 -- 172.21.15.81:0/1162329727 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92740ac9c0 con 0x7f92880736c0 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 -- 172.21.15.81:0/1162329727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 msgr2=0x7f9288073c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 --2- 172.21.15.81:0/1162329727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288073c40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f92740097c0 tx=0x7f927401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 -- 172.21.15.81:0/1162329727 shutdown_connections 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 --2- 172.21.15.81:0/1162329727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288073c40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 -- 172.21.15.81:0/1162329727 >> 172.21.15.81:0/1162329727 conn(0x7f92880cefd0 msgr2=0x7f92880cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 -- 172.21.15.81:0/1162329727 shutdown_connections 2024-05-27T06:54:11.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.838+0000 7f928dc00640 1 -- 172.21.15.81:0/1162329727 wait complete. 2024-05-27T06:54:11.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.842+0000 7f928dc00640 1 Processor -- start 2024-05-27T06:54:11.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.842+0000 7f928dc00640 1 -- start start 2024-05-27T06:54:11.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 0x7f92880c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9288075310 con 0x7f928806d120 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f928806d940 con 0x7f92880736c0 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9288075490 con 0x7f9288156c40 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35044/0 (socket says 172.21.15.81:35044) 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 -- 172.21.15.81:0/238574030 learned_addr learned my addr 172.21.15.81:0/238574030 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286000640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9287400640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 0x7f92880c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:11.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 -- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 msgr2=0x7f92880c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 0x7f92880c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 -- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 msgr2=0x7f9288149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 -- 172.21.15.81:0/238574030 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928815c540 con 0x7f928806d120 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9287400640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 0x7f92880c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:54:11.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f9286a00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f927800c450 tx=0x7f927800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92780052d0 con 0x7f928806d120 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9278005470 con 0x7f928806d120 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9278012850 con 0x7f928806d120 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9288149bb0 con 0x7f928806d120 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f928815bf80 con 0x7f928806d120 2024-05-27T06:54:11.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.846+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928806a560 con 0x7f928806d120 2024-05-27T06:54:11.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.850+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f92780129f0 con 0x7f928806d120 2024-05-27T06:54:11.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.850+0000 7f927ea00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 0x7f9268071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:11.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.850+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f927808f060 con 0x7f928806d120 2024-05-27T06:54:11.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.850+0000 7f9286000640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 0x7f9268071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:11.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.850+0000 7f9286000640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 0x7f9268071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f92740ae000 tx=0x7f9274008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:11.858+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f927805bd20 con 0x7f928806d120 2024-05-27T06:54:12.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.082+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92880c58d0 con 0x7f928806d120 2024-05-27T06:54:12.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f927ea00640 1 -- 172.21.15.81:0/238574030 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6089 (secure 0 0 0) 0x7f927805b620 con 0x7f928806d120 2024-05-27T06:54:12.077 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:12.077 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:08.699129+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,5049,5184]},"id":1}]} 2024-05-27T06:54:12.077 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 44 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 msgr2=0x7f9268071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 0x7f9268071910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f92740ae000 tx=0x7f9274008040 comp rx=0 tx=0).stop 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 msgr2=0x7f928815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f927800c450 tx=0x7f927800c910 comp rx=0 tx=0).stop 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 shutdown_connections 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9288156c40 0x7f92880c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f926806f400 0x7f9268071910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92880736c0 0x7f9288149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:12.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 --2- 172.21.15.81:0/238574030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f928806d120 0x7f928815ac00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:12.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 >> 172.21.15.81:0/238574030 conn(0x7f92880cefd0 msgr2=0x7f9288151110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:12.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 shutdown_connections 2024-05-27T06:54:12.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:12.086+0000 7f928dc00640 1 -- 172.21.15.81:0/238574030 wait complete. 2024-05-27T06:54:12.102 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4302": { "addr": "172.21.15.106:6836/730205609", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6834", "nonce": 730205609, "type": "v2" }, { "addr": "172.21.15.106:6836", "nonce": 730205609, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 5 }, "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:reconnect", "state_seq": 10 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:54:08.699129+0000", "qdb_cluster": [ 4302, 4870, 5049, 5184 ], "qdb_leader": 4302, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4302, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:54:14.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:14.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.278+0000 7f8b31a00640 1 Processor -- start 2024-05-27T06:54:14.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.278+0000 7f8b31a00640 1 -- start start 2024-05-27T06:54:14.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8b2c154970 0x7f8b2c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c0c8410 0x7f8b2c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2c06eec0 con 0x7f8b2c15daf0 2024-05-27T06:54:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2c06f0f0 con 0x7f8b2c154970 2024-05-27T06:54:14.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b31a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2c06f3b0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35046/0 (socket says 172.21.15.81:35046) 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 -- 172.21.15.81:0/3399805765 learned_addr learned my addr 172.21.15.81:0/3399805765 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2b400640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c0c8410 0x7f8b2c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2aa00640 1 --2- 172.21.15.81:0/3399805765 >> v2:172.21.15.106:3300/0 conn(0x7f8b2c154970 0x7f8b2c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 -- 172.21.15.81:0/3399805765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c0c8410 msgr2=0x7f8b2c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:14.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c0c8410 0x7f8b2c06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 -- 172.21.15.81:0/3399805765 >> v2:172.21.15.106:3300/0 conn(0x7f8b2c154970 msgr2=0x7f8b2c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:14.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 --2- 172.21.15.81:0/3399805765 >> v2:172.21.15.106:3300/0 conn(0x7f8b2c154970 0x7f8b2c15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.282+0000 7f8b2a000640 1 -- 172.21.15.81:0/3399805765 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c0cad30 con 0x7f8b2c15daf0 2024-05-27T06:54:14.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b2a000640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0704c0 tx=0x7f8b1801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c1401d6dae68a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:14.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b29600640 1 -- 172.21.15.81:0/3399805765 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180b0070 con 0x7f8b2c15daf0 2024-05-27T06:54:14.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b29600640 1 -- 172.21.15.81:0/3399805765 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b180ab600 con 0x7f8b2c15daf0 2024-05-27T06:54:14.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b29600640 1 -- 172.21.15.81:0/3399805765 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180ab9d0 con 0x7f8b2c15daf0 2024-05-27T06:54:14.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 -- 172.21.15.81:0/3399805765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c15daf0 msgr2=0x7f8b2c0c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c0704c0 tx=0x7f8b1801c260 comp rx=0 tx=0).stop 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 -- 172.21.15.81:0/3399805765 shutdown_connections 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c0c8410 0x7f8b2c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 --2- 172.21.15.81:0/3399805765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c15daf0 0x7f8b2c0c7ed0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 --2- 172.21.15.81:0/3399805765 >> v2:172.21.15.106:3300/0 conn(0x7f8b2c154970 0x7f8b2c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 -- 172.21.15.81:0/3399805765 >> 172.21.15.81:0/3399805765 conn(0x7f8b2c0cb2e0 msgr2=0x7f8b2c0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 -- 172.21.15.81:0/3399805765 shutdown_connections 2024-05-27T06:54:14.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.286+0000 7f8b31a00640 1 -- 172.21.15.81:0/3399805765 wait complete. 2024-05-27T06:54:14.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.290+0000 7f8b31a00640 1 Processor -- start 2024-05-27T06:54:14.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.290+0000 7f8b31a00640 1 -- start start 2024-05-27T06:54:14.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b2c154970 0x7f8b2c1f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c15daf0 0x7f8b2c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c0c9520 con 0x7f8b2c0c8410 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c0c93a0 con 0x7f8b2c154970 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2c15df00 con 0x7f8b2c15daf0 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35050/0 (socket says 172.21.15.81:35050) 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 -- 172.21.15.81:0/946900998 learned_addr learned my addr 172.21.15.81:0/946900998 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2b400640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c15daf0 0x7f8b2c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2a000640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b2c154970 0x7f8b2c1f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 -- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c15daf0 msgr2=0x7f8b2c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:14.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c15daf0 0x7f8b2c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 -- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b2c154970 msgr2=0x7f8b2c1f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:14.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b2c154970 0x7f8b2c1f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 -- 172.21.15.81:0/946900998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c0c6c50 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b2aa00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00d600 tx=0x7f8b1c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c004240 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1c0043e0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2c1f5e40 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c003840 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2c1f63e0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.294+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b2c06a560 con 0x7f8b2c0c8410 2024-05-27T06:54:14.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.298+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8b1c0039e0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.298+0000 7f8b22a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 0x7f8b0c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:14.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.298+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f8b1c08ff90 con 0x7f8b2c0c8410 2024-05-27T06:54:14.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.298+0000 7f8b2a000640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 0x7f8b0c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:14.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.298+0000 7f8b2a000640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 0x7f8b0c071940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b180ad000 tx=0x7f8b180a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:14.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.302+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8b1c05c6f0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.538+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b2c06fa90 con 0x7f8b2c0c8410 2024-05-27T06:54:14.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.538+0000 7f8b22a00640 1 -- 172.21.15.81:0/946900998 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6086 (secure 0 0 0) 0x7f8b1c0609f0 con 0x7f8b2c0c8410 2024-05-27T06:54:14.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:14.532 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:13.716934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4870,4302]},"id":1}]} 2024-05-27T06:54:14.532 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 45 2024-05-27T06:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 msgr2=0x7f8b0c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 0x7f8b0c071940 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b180ad000 tx=0x7f8b180a6000 comp rx=0 tx=0).stop 2024-05-27T06:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 msgr2=0x7f8b2c0c6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00d600 tx=0x7f8b1c00dac0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 shutdown_connections 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b2c15daf0 0x7f8b2c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b0c06f430 0x7f8b0c071940 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b2c154970 0x7f8b2c1f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 --2- 172.21.15.81:0/946900998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0c8410 0x7f8b2c0c6710 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 >> 172.21.15.81:0/946900998 conn(0x7f8b2c0cb2e0 msgr2=0x7f8b2c14d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 shutdown_connections 2024-05-27T06:54:14.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:14.542+0000 7f8b31a00640 1 -- 172.21.15.81:0/946900998 wait complete. 2024-05-27T06:54:14.555 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:16.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:16.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.746+0000 7f01b4600640 1 Processor -- start 2024-05-27T06:54:16.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.746+0000 7f01b4600640 1 -- start start 2024-05-27T06:54:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b4600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f01ac0c9770 0x7f01ac0cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b4600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f01ac160590 0x7f01ac06e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b4600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01ac0d06f0 con 0x7f01ac06e290 2024-05-27T06:54:16.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b4600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f01ac15d5e0 con 0x7f01ac160590 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b4600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f01ac15c9d0 con 0x7f01ac0c9770 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f01ac160590 0x7f01ac06e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f01ac160590 0x7f01ac06e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42706/0 (socket says 172.21.15.81:42706) 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 -- 172.21.15.81:0/304508202 learned_addr learned my addr 172.21.15.81:0/304508202 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b1c00640 1 --2- 172.21.15.81:0/304508202 >> v2:172.21.15.106:3300/0 conn(0x7f01ac0c9770 0x7f01ac0cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 -- 172.21.15.81:0/304508202 >> v2:172.21.15.106:3300/0 conn(0x7f01ac0c9770 msgr2=0x7f01ac0cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 --2- 172.21.15.81:0/304508202 >> v2:172.21.15.106:3300/0 conn(0x7f01ac0c9770 0x7f01ac0cfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 -- 172.21.15.81:0/304508202 >> v1:172.21.15.106:6789/0 conn(0x7f01ac06e290 legacy=0x7f01ac15fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.750+0000 7f01b2600640 1 -- 172.21.15.81:0/304508202 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac15ccb0 con 0x7f01ac160590 2024-05-27T06:54:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b2600640 1 --2- 172.21.15.81:0/304508202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac06e6a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0d09c0 tx=0x7f01a401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f00a4e78a9326fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01abe00640 1 -- 172.21.15.81:0/304508202 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a40a5710 con 0x7f01ac160590 2024-05-27T06:54:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01abe00640 1 -- 172.21.15.81:0/304508202 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a40a58b0 con 0x7f01ac160590 2024-05-27T06:54:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01abe00640 1 -- 172.21.15.81:0/304508202 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a40a5c80 con 0x7f01ac160590 2024-05-27T06:54:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 -- 172.21.15.81:0/304508202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 msgr2=0x7f01ac06e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 --2- 172.21.15.81:0/304508202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac06e6a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0d09c0 tx=0x7f01a401c960 comp rx=0 tx=0).stop 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 -- 172.21.15.81:0/304508202 shutdown_connections 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 --2- 172.21.15.81:0/304508202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac06e6a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 --2- 172.21.15.81:0/304508202 >> v2:172.21.15.106:3300/0 conn(0x7f01ac0c9770 0x7f01ac0cfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 -- 172.21.15.81:0/304508202 >> 172.21.15.81:0/304508202 conn(0x7f01ac068ab0 msgr2=0x7f01ac068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 -- 172.21.15.81:0/304508202 shutdown_connections 2024-05-27T06:54:16.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.754+0000 7f01b4600640 1 -- 172.21.15.81:0/304508202 wait complete. 2024-05-27T06:54:16.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 Processor -- start 2024-05-27T06:54:16.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 -- start start 2024-05-27T06:54:16.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac06e290 0x7f01ac0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0c9770 0x7f01ac0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac0d04b0 con 0x7f01ac160590 2024-05-27T06:54:16.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac0d0330 con 0x7f01ac0c9770 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b4600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f01ac169910 con 0x7f01ac06e290 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42708/0 (socket says 172.21.15.81:42708) 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b2600640 1 -- 172.21.15.81:0/3231459858 learned_addr learned my addr 172.21.15.81:0/3231459858 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b1200640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0c9770 0x7f01ac0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.758+0000 7f01b1c00640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac06e290 0x7f01ac0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 -- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac06e290 msgr2=0x7f01ac0cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac06e290 0x7f01ac0cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 -- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0c9770 msgr2=0x7f01ac0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0c9770 0x7f01ac0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 -- 172.21.15.81:0/3231459858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac0caf90 con 0x7f01ac160590 2024-05-27T06:54:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b2600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f01a40af040 tx=0x7f01a4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a4009510 con 0x7f01ac160590 2024-05-27T06:54:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a40be070 con 0x7f01ac160590 2024-05-27T06:54:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac1f6790 con 0x7f01ac160590 2024-05-27T06:54:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a40b88d0 con 0x7f01ac160590 2024-05-27T06:54:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac0cb510 con 0x7f01ac160590 2024-05-27T06:54:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01ac06a560 con 0x7f01ac160590 2024-05-27T06:54:16.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.762+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f01a40b8a70 con 0x7f01ac160590 2024-05-27T06:54:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.766+0000 7f01a9600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 0x7f018c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.766+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f01a4129e30 con 0x7f01ac160590 2024-05-27T06:54:16.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.766+0000 7f01b1c00640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 0x7f018c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:16.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.766+0000 7f01b1c00640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 0x7f018c071940 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009ac0 tx=0x7f01a0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:16.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.770+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f01a40c7020 con 0x7f01ac160590 2024-05-27T06:54:16.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.994+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01ac073050 con 0x7f01ac160590 2024-05-27T06:54:16.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.994+0000 7f01a9600640 1 -- 172.21.15.81:0/3231459858 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6086 (secure 0 0 0) 0x7f01a40165e0 con 0x7f01ac160590 2024-05-27T06:54:16.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:16.987 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:13.716934+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4870,4302]},"id":1}]} 2024-05-27T06:54:16.988 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 45 2024-05-27T06:54:16.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 msgr2=0x7f018c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 0x7f018c071940 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009ac0 tx=0x7f01a0009190 comp rx=0 tx=0).stop 2024-05-27T06:54:16.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 msgr2=0x7f01ac0c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f01a40af040 tx=0x7f01a4019700 comp rx=0 tx=0).stop 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 shutdown_connections 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f018c06f430 0x7f018c071940 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01ac160590 0x7f01ac0c4220 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01ac0c9770 0x7f01ac0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 --2- 172.21.15.81:0/3231459858 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f01ac06e290 0x7f01ac0cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 >> 172.21.15.81:0/3231459858 conn(0x7f01ac068ab0 msgr2=0x7f01ac146f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 shutdown_connections 2024-05-27T06:54:16.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:16.998+0000 7f01b4600640 1 -- 172.21.15.81:0/3231459858 wait complete. 2024-05-27T06:54:17.018 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:54:19.019 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:19.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 Processor -- start 2024-05-27T06:54:19.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 -- start start 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdeec0ca1e0 0x7fdeec1569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec15ced0 0x7fdeec15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdeec156ee0 con 0x7fdeec0724b0 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fdeec160130 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef3000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdeec160410 con 0x7fdeec15ced0 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef1c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdeec0ca1e0 0x7fdeec1569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef1c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdeec0ca1e0 0x7fdeec1569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42710/0 (socket says 172.21.15.81:42710) 2024-05-27T06:54:19.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.186+0000 7fdef1c00640 1 -- 172.21.15.81:0/4034982042 learned_addr learned my addr 172.21.15.81:0/4034982042 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1200640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec15ced0 0x7fdeec15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1c00640 1 -- 172.21.15.81:0/4034982042 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec15ced0 msgr2=0x7fdeec15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1c00640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec15ced0 0x7fdeec15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1c00640 1 -- 172.21.15.81:0/4034982042 >> v1:172.21.15.106:6789/0 conn(0x7fdeec0724b0 legacy=0x7fdeec0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:19.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1c00640 1 -- 172.21.15.81:0/4034982042 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec1606f0 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.190+0000 7fdef1c00640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec1569a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009870 tx=0x7fdee001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9ad96e9809c6c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdeebe00640 1 -- 172.21.15.81:0/4034982042 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee00b5070 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdeebe00640 1 -- 172.21.15.81:0/4034982042 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee00a89d0 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdeebe00640 1 -- 172.21.15.81:0/4034982042 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee00a8da0 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- 172.21.15.81:0/4034982042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 msgr2=0x7fdeec1569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:19.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec1569a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009870 tx=0x7fdee001ed50 comp rx=0 tx=0).stop 2024-05-27T06:54:19.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- 172.21.15.81:0/4034982042 shutdown_connections 2024-05-27T06:54:19.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec15ced0 0x7fdeec15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 --2- 172.21.15.81:0/4034982042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec1569a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- 172.21.15.81:0/4034982042 >> 172.21.15.81:0/4034982042 conn(0x7fdeec068880 msgr2=0x7fdeec068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:19.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- 172.21.15.81:0/4034982042 shutdown_connections 2024-05-27T06:54:19.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- 172.21.15.81:0/4034982042 wait complete. 2024-05-27T06:54:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 Processor -- start 2024-05-27T06:54:19.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.194+0000 7fdef3000640 1 -- start start 2024-05-27T06:54:19.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec0724b0 0x7fdeec150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 0x7fdeec1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec0ca820 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdeec15c9b0 con 0x7fdeec15ced0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdeec0ca9a0 con 0x7fdeec0724b0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42726/0 (socket says 172.21.15.81:42726) 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 -- 172.21.15.81:0/1425099678 learned_addr learned my addr 172.21.15.81:0/1425099678 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1c00640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec0724b0 0x7fdeec150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef2600640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 0x7fdeec1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 -- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec0724b0 msgr2=0x7fdeec150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec0724b0 0x7fdeec150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 -- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 msgr2=0x7fdeec1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 0x7fdeec1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 -- 172.21.15.81:0/1425099678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec151070 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef2600640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 0x7fdeec1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef1200640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fded800d620 tx=0x7fded800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded800b620 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded8019070 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeec15a790 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded800b9e0 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeec150610 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.198+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeec050870 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.202+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fded800bb80 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.202+0000 7fdee9600640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 0x7fdecc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:19.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.202+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fded808f940 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.202+0000 7fdef1c00640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 0x7fdecc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:19.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.202+0000 7fdef1c00640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 0x7fdecc071940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdee00b2000 tx=0x7fdee0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:19.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.206+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fded805c570 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.434+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdeec0cab90 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.434+0000 7fdee9600640 1 -- 172.21.15.81:0/1425099678 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6091 (secure 0 0 0) 0x7fded805be70 con 0x7fdeec0ca1e0 2024-05-27T06:54:19.427 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:19.427 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[4302,4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:54:19.428 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 46 2024-05-27T06:54:19.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 msgr2=0x7fdecc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:19.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 0x7fdecc071940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdee00b2000 tx=0x7fdee0009040 comp rx=0 tx=0).stop 2024-05-27T06:54:19.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 msgr2=0x7fdeec154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fded800d620 tx=0x7fded800dae0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 shutdown_connections 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdecc06f430 0x7fdecc071940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdeec15ced0 0x7fdeec1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdeec0ca1e0 0x7fdeec154740 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 --2- 172.21.15.81:0/1425099678 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdeec0724b0 0x7fdeec150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 >> 172.21.15.81:0/1425099678 conn(0x7fdeec068880 msgr2=0x7fdeec153780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 shutdown_connections 2024-05-27T06:54:19.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:19.438+0000 7fdef3000640 1 -- 172.21.15.81:0/1425099678 wait complete. 2024-05-27T06:54:19.451 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=1, replacing gid=4287 2024-05-27T06:54:19.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2024-05-27T06:54:26.451 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-05-27T06:54:26.452 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-27T06:54:26.452 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-27T06:54:26.455 INFO:tasks.ceph.mds.a:Started 2024-05-27T06:54:26.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:54:26.563 INFO:tasks.ceph.mds.a.smithi081.stdout:starting mds.a at 2024-05-27T06:54:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.618+0000 7f04b1000640 1 Processor -- start 2024-05-27T06:54:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.618+0000 7f04b1000640 1 -- start start 2024-05-27T06:54:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04b1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04b1000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04b1000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac0c9f80 con 0x7f04ac15cb80 2024-05-27T06:54:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04b1000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac0c58b0 con 0x7f04ac0c8880 2024-05-27T06:54:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04b1000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac0c83f0 con 0x7f04ac06c790 2024-05-27T06:54:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04aa800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a3e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04aa800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:40262/0 (socket says 172.21.15.81:40262) 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04aa800640 1 -- 172.21.15.81:0/3873304298 learned_addr learned my addr 172.21.15.81:0/3873304298 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a9e00640 1 --1- 172.21.15.81:0/3873304298 >> v1:172.21.15.106:6789/0 conn(0x7f04ac15cb80 0x7f04ac152cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:48976/0 (socket says 172.21.15.81:48976) 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a3e00640 1 -- 172.21.15.81:0/3873304298 >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 msgr2=0x7f04ac0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a3e00640 1 --2- 172.21.15.81:0/3873304298 >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a3e00640 1 -- 172.21.15.81:0/3873304298 >> v1:172.21.15.106:6789/0 conn(0x7f04ac15cb80 legacy=0x7f04ac152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04a3e00640 1 -- 172.21.15.81:0/3873304298 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0c52f0 con 0x7f04ac06c790 2024-05-27T06:54:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.622+0000 7f04aa800640 1 --2- 172.21.15.81:0/3873304298 >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:54:26.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04a3e00640 1 --2- 172.21.15.81:0/3873304298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac06cba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f049400ba60 tx=0x7f049401eb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ccf58b04e6cad2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04a9400640 1 -- 172.21.15.81:0/3873304298 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04940b4070 con 0x7f04ac06c790 2024-05-27T06:54:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04a9400640 1 -- 172.21.15.81:0/3873304298 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04940a79c0 con 0x7f04ac06c790 2024-05-27T06:54:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04a9400640 1 -- 172.21.15.81:0/3873304298 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04940a7d90 con 0x7f04ac06c790 2024-05-27T06:54:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- 172.21.15.81:0/3873304298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 msgr2=0x7f04ac06cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 --2- 172.21.15.81:0/3873304298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac06cba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f049400ba60 tx=0x7f049401eb70 comp rx=0 tx=0).stop 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- 172.21.15.81:0/3873304298 shutdown_connections 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 --2- 172.21.15.81:0/3873304298 >> v2:172.21.15.106:3300/0 conn(0x7f04ac0c8880 0x7f04ac0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 --2- 172.21.15.81:0/3873304298 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac06cba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- 172.21.15.81:0/3873304298 >> 172.21.15.81:0/3873304298 conn(0x7f04ac070700 msgr2=0x7f04ac0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- 172.21.15.81:0/3873304298 shutdown_connections 2024-05-27T06:54:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- 172.21.15.81:0/3873304298 wait complete. 2024-05-27T06:54:26.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 Processor -- start 2024-05-27T06:54:26.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.626+0000 7f04b1000640 1 -- start start 2024-05-27T06:54:26.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ac0c8880 0x7f04ac0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac0c9c80 con 0x7f04ac15cb80 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac0c9b00 con 0x7f04ac0c8880 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac1654f0 con 0x7f04ac06c790 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38868/0 (socket says 172.21.15.81:38868) 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 -- 172.21.15.81:0/2061609725 learned_addr learned my addr 172.21.15.81:0/2061609725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04a9e00640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04a3e00640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ac0c8880 0x7f04ac0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 -- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 msgr2=0x7f04ac148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac148520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 -- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ac0c8880 msgr2=0x7f04ac0cc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ac0c8880 0x7f04ac0cc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 -- 172.21.15.81:0/2061609725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac148a60 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04aa800640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f04a400ef90 tx=0x7f04a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a400cac0 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a400cc60 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac147d60 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4003bc0 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac1f5bd0 con 0x7f04ac15cb80 2024-05-27T06:54:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.630+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac06a560 con 0x7f04ac15cb80 2024-05-27T06:54:26.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.634+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f04a401b020 con 0x7f04ac15cb80 2024-05-27T06:54:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.634+0000 7f04a2000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 0x7f0488071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:54:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.634+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f04a408f5d0 con 0x7f04ac15cb80 2024-05-27T06:54:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.634+0000 7f04a9e00640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 0x7f0488071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:54:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.634+0000 7f04a9e00640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 0x7f0488071940 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f049c000f70 tx=0x7f049c007e90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:54:26.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.638+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f04a4022020 con 0x7f04ac15cb80 2024-05-27T06:54:26.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.834+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04ac15c860 con 0x7f04ac15cb80 2024-05-27T06:54:26.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.834+0000 7f04a2000640 1 -- 172.21.15.81:0/2061609725 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7f04a4010040 con 0x7f04ac15cb80 2024-05-27T06:54:26.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:54:26.826 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:54:26.826 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:54:26.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 msgr2=0x7f0488071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 0x7f0488071940 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f049c000f70 tx=0x7f049c007e90 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 msgr2=0x7f04ac13e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f04a400ef90 tx=0x7f04a400c490 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 shutdown_connections 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f048806f430 0x7f0488071940 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ac15cb80 0x7f04ac13e000 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f04ac0c8880 0x7f04ac0cc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 --2- 172.21.15.81:0/2061609725 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f04ac06c790 0x7f04ac148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 >> 172.21.15.81:0/2061609725 conn(0x7f04ac070700 msgr2=0x7f04ac148f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 shutdown_connections 2024-05-27T06:54:26.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:54:26.838+0000 7f04b1000640 1 -- 172.21.15.81:0/2061609725 wait complete. 2024-05-27T06:54:26.847 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-05-27T06:54:26.848 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2024-05-27T06:54:34.314 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:54:34.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:55:04.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:55:04.400 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:55:34.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:55:34.460 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:55:37.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:38.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.072+0000 7fcfe4c00640 1 Processor -- start 2024-05-27T06:55:38.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.072+0000 7fcfe4c00640 1 -- start start 2024-05-27T06:55:38.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfe4c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcfe0154780 0x7fcfe0072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfe4c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfe4c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00c9c70 con 0x7fcfe00c9630 2024-05-27T06:55:38.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfe4c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe00c9a40 con 0x7fcfe0154780 2024-05-27T06:55:38.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfe4c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe015b0c0 con 0x7fcfe0072a60 2024-05-27T06:55:38.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfd7e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fcfe0154780 0x7fcfe0072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:39242/0 (socket says 172.21.15.81:39242) 2024-05-27T06:55:38.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 -- 172.21.15.81:0/1589813746 learned_addr learned my addr 172.21.15.81:0/1589813746 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:38.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 -- 172.21.15.81:0/1589813746 >> v2:172.21.15.106:3300/0 conn(0x7fcfe0154780 msgr2=0x7fcfe0072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:38.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 --2- 172.21.15.81:0/1589813746 >> v2:172.21.15.106:3300/0 conn(0x7fcfe0154780 0x7fcfe0072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 -- 172.21.15.81:0/1589813746 >> v1:172.21.15.106:6789/0 conn(0x7fcfe00c9630 legacy=0x7fcfe015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.076+0000 7fcfdfe00640 1 -- 172.21.15.81:0/1589813746 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe015a4d0 con 0x7fcfe0072a60 2024-05-27T06:55:38.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfdfe00640 1 --2- 172.21.15.81:0/1589813746 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009e50 tx=0x7fcfd801ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62d81ccf2963451 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfdea00640 1 -- 172.21.15.81:0/1589813746 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80090d0 con 0x7fcfe0072a60 2024-05-27T06:55:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfdea00640 1 -- 172.21.15.81:0/1589813746 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd80a5810 con 0x7fcfe0072a60 2024-05-27T06:55:38.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfdea00640 1 -- 172.21.15.81:0/1589813746 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8009270 con 0x7fcfe0072a60 2024-05-27T06:55:38.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 -- 172.21.15.81:0/1589813746 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 msgr2=0x7fcfe0164e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/1589813746 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009e50 tx=0x7fcfd801ca50 comp rx=0 tx=0).stop 2024-05-27T06:55:38.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 -- 172.21.15.81:0/1589813746 shutdown_connections 2024-05-27T06:55:38.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/1589813746 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe0164e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/1589813746 >> v2:172.21.15.106:3300/0 conn(0x7fcfe0154780 0x7fcfe0072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 -- 172.21.15.81:0/1589813746 >> 172.21.15.81:0/1589813746 conn(0x7fcfe0068880 msgr2=0x7fcfe0068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:38.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 -- 172.21.15.81:0/1589813746 shutdown_connections 2024-05-27T06:55:38.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.080+0000 7fcfe4c00640 1 -- 172.21.15.81:0/1589813746 wait complete. 2024-05-27T06:55:38.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 Processor -- start 2024-05-27T06:55:38.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 -- start start 2024-05-27T06:55:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe01576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 0x7fcfe01f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 0x7fcfe0161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe00cd280 con 0x7fcfe00c9630 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe015ad50 con 0x7fcfe0154780 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe015e1d0 con 0x7fcfe0072a60 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 0x7fcfe01f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 0x7fcfe01f15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56856/0 (socket says 172.21.15.81:56856) 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7e00640 1 -- 172.21.15.81:0/949924122 learned_addr learned my addr 172.21.15.81:0/949924122 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:38.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe01576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdfe00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 0x7fcfe0161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 -- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 msgr2=0x7fcfe0161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 0x7fcfe0161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 -- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 msgr2=0x7fcfe01f15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 0x7fcfe01f15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 -- 172.21.15.81:0/949924122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe01f1b10 con 0x7fcfe0072a60 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdfe00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 0x7fcfe0161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfdf400640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe01576a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0000f70 tx=0x7fcfd000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0017070 con 0x7fcfe0072a60 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd00052f0 con 0x7fcfe0072a60 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0003840 con 0x7fcfe0072a60 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.084+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe015eb50 con 0x7fcfe0072a60 2024-05-27T06:55:38.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe015f110 con 0x7fcfe0072a60 2024-05-27T06:55:38.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe0050870 con 0x7fcfe0072a60 2024-05-27T06:55:38.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcfd001a070 con 0x7fcfe0072a60 2024-05-27T06:55:38.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfd7400640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 0x7fcfbc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fcfd008e670 con 0x7fcfe0072a60 2024-05-27T06:55:38.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.088+0000 7fcfd7e00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 0x7fcfbc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.092+0000 7fcfd7e00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 0x7fcfbc071910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcfe01588d0 tx=0x7fcfc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.096+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fcfd005ad70 con 0x7fcfe0072a60 2024-05-27T06:55:38.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.404+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfe00cbd60 con 0x7fcfe0072a60 2024-05-27T06:55:38.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfd7400640 1 -- 172.21.15.81:0/949924122 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7fcfd00633b0 con 0x7fcfe0072a60 2024-05-27T06:55:38.401 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:38.401 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:38.401 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 msgr2=0x7fcfbc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 0x7fcfbc071910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcfe01588d0 tx=0x7fcfc8008040 comp rx=0 tx=0).stop 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 msgr2=0x7fcfe01576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe01576a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0000f70 tx=0x7fcfd000c800 comp rx=0 tx=0).stop 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 shutdown_connections 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcfbc06f400 0x7fcfbc071910 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfe0154780 0x7fcfe0161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfe00c9630 0x7fcfe01f15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 --2- 172.21.15.81:0/949924122 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fcfe0072a60 0x7fcfe01576a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 >> 172.21.15.81:0/949924122 conn(0x7fcfe0068880 msgr2=0x7fcfe0068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:38.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 shutdown_connections 2024-05-27T06:55:38.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.408+0000 7fcfe4c00640 1 -- 172.21.15.81:0/949924122 wait complete. 2024-05-27T06:55:38.426 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2024-05-27T06:55:38.426 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-05-27T06:55:38.426 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T06:55:38.445 INFO:tasks.ceph.mds.f:Stopped 2024-05-27T06:55:38.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.596+0000 7f3ae7000640 1 Processor -- start 2024-05-27T06:55:38.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 -- start start 2024-05-27T06:55:38.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00ca770 0x7f3ae00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ae00c78b0 con 0x7f3ae00d02f0 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3ae00c6ee0 con 0x7f3ae00cbd00 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae7000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae00c71c0 con 0x7f3ae00ca770 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae5000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3ae00d02f0 0x7f3ae0154200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43326/0 (socket says 172.21.15.81:43326) 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56870/0 (socket says 172.21.15.81:56870) 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3ae5000640 1 -- 172.21.15.81:0/3699913894 learned_addr learned my addr 172.21.15.81:0/3699913894 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adfe00640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00ca770 0x7f3ae00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 -- 172.21.15.81:0/3699913894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00ca770 msgr2=0x7f3ae00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00ca770 0x7f3ae00cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 -- 172.21.15.81:0/3699913894 >> v1:172.21.15.81:6789/0 conn(0x7f3ae00d02f0 legacy=0x7f3ae0154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adf400640 1 -- 172.21.15.81:0/3699913894 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae00c7b70 con 0x7f3ae00cbd00 2024-05-27T06:55:38.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.600+0000 7f3adea00640 1 -- 172.21.15.81:0/3699913894 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588040785 0 0) 0x7f3ae00c78b0 con 0x7f3ae00d02f0 2024-05-27T06:55:38.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3adf400640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0097c0 tx=0x7f3acc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78afaeb4c931251 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3adea00640 1 -- 172.21.15.81:0/3699913894 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0b1070 con 0x7f3ae00cbd00 2024-05-27T06:55:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3adea00640 1 -- 172.21.15.81:0/3699913894 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3acc0ac410 con 0x7f3ae00cbd00 2024-05-27T06:55:38.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3adea00640 1 -- 172.21.15.81:0/3699913894 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc0ac7e0 con 0x7f3ae00cbd00 2024-05-27T06:55:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 -- 172.21.15.81:0/3699913894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00cbd00 msgr2=0x7f3ae00cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0097c0 tx=0x7f3acc01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 -- 172.21.15.81:0/3699913894 shutdown_connections 2024-05-27T06:55:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cfdb0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 --2- 172.21.15.81:0/3699913894 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00ca770 0x7f3ae00cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 -- 172.21.15.81:0/3699913894 >> 172.21.15.81:0/3699913894 conn(0x7f3ae0068ae0 msgr2=0x7f3ae0068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 -- 172.21.15.81:0/3699913894 shutdown_connections 2024-05-27T06:55:38.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.604+0000 7f3ae7000640 1 -- 172.21.15.81:0/3699913894 wait complete. 2024-05-27T06:55:38.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 Processor -- start 2024-05-27T06:55:38.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 -- start start 2024-05-27T06:55:38.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00d02f0 0x7f3ae01493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0167590 con 0x7f3ae00ca770 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0154b60 con 0x7f3ae00cbd00 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae7000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae0167710 con 0x7f3ae00d02f0 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56872/0 (socket says 172.21.15.81:56872) 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 -- 172.21.15.81:0/984705813 learned_addr learned my addr 172.21.15.81:0/984705813 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adf400640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ae5000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00d02f0 0x7f3ae01493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 -- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00d02f0 msgr2=0x7f3ae01493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00d02f0 0x7f3ae01493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 -- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae00cbd00 msgr2=0x7f3ae00cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 -- 172.21.15.81:0/984705813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae0163e90 con 0x7f3ae00ca770 2024-05-27T06:55:38.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3adfe00640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000e870 tx=0x7f3ad000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.608+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0017070 con 0x7f3ae00ca770 2024-05-27T06:55:38.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad000c940 con 0x7f3ae00ca770 2024-05-27T06:55:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad000ccf0 con 0x7f3ae00ca770 2024-05-27T06:55:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae006e660 con 0x7f3ae00ca770 2024-05-27T06:55:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae0164810 con 0x7f3ae00ca770 2024-05-27T06:55:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae006a560 con 0x7f3ae00ca770 2024-05-27T06:55:38.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3ad001a070 con 0x7f3ae00ca770 2024-05-27T06:55:38.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.612+0000 7f3ad7e00640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 0x7f3ac0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:38.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.616+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f3ad008f2b0 con 0x7f3ae00ca770 2024-05-27T06:55:38.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.616+0000 7f3adf400640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 0x7f3ac0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:38.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.616+0000 7f3adf400640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 0x7f3ac0071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ae000 tx=0x7f3acc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:38.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.620+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3ad005b9c0 con 0x7f3ae00ca770 2024-05-27T06:55:38.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.816+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ae0072280 con 0x7f3ae00ca770 2024-05-27T06:55:38.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.816+0000 7f3ad7e00640 1 -- 172.21.15.81:0/984705813 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7f3ad005fcc0 con 0x7f3ae00ca770 2024-05-27T06:55:38.812 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:38.812 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:38.812 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:38.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 msgr2=0x7f3ac0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 0x7f3ac0071910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3acc0ae000 tx=0x7f3acc008040 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 msgr2=0x7f3ae0163950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000e870 tx=0x7f3ad000ec70 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 shutdown_connections 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3ae00d02f0 0x7f3ae01493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3ac006f400 0x7f3ac0071910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ae00cbd00 0x7f3ae00cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 --2- 172.21.15.81:0/984705813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae00ca770 0x7f3ae0163950 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 >> 172.21.15.81:0/984705813 conn(0x7f3ae0068ae0 msgr2=0x7f3ae0150300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 shutdown_connections 2024-05-27T06:55:38.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:38.820+0000 7f3ae7000640 1 -- 172.21.15.81:0/984705813 wait complete. 2024-05-27T06:55:38.840 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:40.842 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:41.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.052+0000 7f9d33400640 1 Processor -- start 2024-05-27T06:55:41.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.052+0000 7f9d33400640 1 -- start start 2024-05-27T06:55:41.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d33400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9d2c0c6de0 0x7f9d2c0cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d33400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9d2c15a3e0 0x7f9d2c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d33400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d2c0755a0 con 0x7f9d2c0cc930 2024-05-27T06:55:41.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d33400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2c0c7400 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d33400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9d2c0c98c0 con 0x7f9d2c15a3e0 2024-05-27T06:55:41.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d31400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9d2c0cc930 0x7f9d2c0cf890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43330/0 (socket says 172.21.15.81:43330) 2024-05-27T06:55:41.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d31400640 1 -- 172.21.15.81:0/968606626 learned_addr learned my addr 172.21.15.81:0/968606626 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:55:41.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d30a00640 1 --2- 172.21.15.81:0/968606626 >> v2:172.21.15.81:3300/0 conn(0x7f9d2c0c6de0 0x7f9d2c0cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d2be00640 1 --2- 172.21.15.81:0/968606626 >> v2:172.21.15.106:3300/0 conn(0x7f9d2c15a3e0 0x7f9d2c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d2b400640 1 -- 172.21.15.81:0/968606626 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421668819 0 0) 0x7f9d2c0755a0 con 0x7f9d2c0cc930 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d2b400640 1 -- 172.21.15.81:0/968606626 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d14003530 con 0x7f9d2c0cc930 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d30a00640 1 -- 172.21.15.81:0/968606626 >> v2:172.21.15.106:3300/0 conn(0x7f9d2c15a3e0 msgr2=0x7f9d2c0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d30a00640 1 --2- 172.21.15.81:0/968606626 >> v2:172.21.15.106:3300/0 conn(0x7f9d2c15a3e0 0x7f9d2c0ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d30a00640 1 -- 172.21.15.81:0/968606626 >> v1:172.21.15.81:6789/0 conn(0x7f9d2c0cc930 legacy=0x7f9d2c0cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.056+0000 7f9d30a00640 1 -- 172.21.15.81:0/968606626 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2c0c9ba0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d30a00640 1 --2- 172.21.15.81:0/968606626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c0cbd40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c074110 tx=0x7f9d2001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56f7a259dbdda59 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d2b400640 1 -- 172.21.15.81:0/968606626 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200b5070 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d2b400640 1 -- 172.21.15.81:0/968606626 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d200a8970 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d2b400640 1 -- 172.21.15.81:0/968606626 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200a8d40 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 -- 172.21.15.81:0/968606626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 msgr2=0x7f9d2c0cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 --2- 172.21.15.81:0/968606626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c0cbd40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c074110 tx=0x7f9d2001ec70 comp rx=0 tx=0).stop 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 -- 172.21.15.81:0/968606626 shutdown_connections 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 --2- 172.21.15.81:0/968606626 >> v2:172.21.15.106:3300/0 conn(0x7f9d2c15a3e0 0x7f9d2c0ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 --2- 172.21.15.81:0/968606626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c0cbd40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 -- 172.21.15.81:0/968606626 >> 172.21.15.81:0/968606626 conn(0x7f9d2c068ab0 msgr2=0x7f9d2c068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:41.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 -- 172.21.15.81:0/968606626 shutdown_connections 2024-05-27T06:55:41.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.060+0000 7f9d33400640 1 -- 172.21.15.81:0/968606626 wait complete. 2024-05-27T06:55:41.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.064+0000 7f9d33400640 1 Processor -- start 2024-05-27T06:55:41.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.064+0000 7f9d33400640 1 -- start start 2024-05-27T06:55:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d2c0cc930 0x7f9d2c1f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d2c15a3e0 0x7f9d2c14b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2c0c95c0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2c15a7f0 con 0x7f9d2c0cc930 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9d2c0ca820 con 0x7f9d2c15a3e0 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56894/0 (socket says 172.21.15.81:56894) 2024-05-27T06:55:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 -- 172.21.15.81:0/3896396222 learned_addr learned my addr 172.21.15.81:0/3896396222 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d2be00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d2c0cc930 0x7f9d2c1f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d31400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d2c15a3e0 0x7f9d2c14b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 -- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d2c15a3e0 msgr2=0x7f9d2c14b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d2c15a3e0 0x7f9d2c14b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 -- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d2c0cc930 msgr2=0x7f9d2c1f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d2c0cc930 0x7f9d2c1f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 -- 172.21.15.81:0/3896396222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2c1f4210 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d30a00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d200b3040 tx=0x7f9d2001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200bf070 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d200b0660 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d2c1f44f0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d200b5040 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2c1f4ab0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.068+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d080052a0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.072+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9d200c2070 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.072+0000 7f9d28c00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 0x7f9d0c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:41.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.072+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f9d2012b870 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.072+0000 7f9d2be00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 0x7f9d0c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:41.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.076+0000 7f9d2be00640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 0x7f9d0c071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c151430 tx=0x7f9d18009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:41.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.084+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9d200187d0 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.388+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d08005860 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.388+0000 7f9d28c00640 1 -- 172.21.15.81:0/3896396222 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7f9d200f8010 con 0x7f9d2c0c6de0 2024-05-27T06:55:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:41.383 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:41.384 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 msgr2=0x7f9d0c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 0x7f9d0c071910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c151430 tx=0x7f9d18009190 comp rx=0 tx=0).stop 2024-05-27T06:55:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 msgr2=0x7f9d2c14e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:41.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9d200b3040 tx=0x7f9d2001ba10 comp rx=0 tx=0).stop 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 shutdown_connections 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9d2c15a3e0 0x7f9d2c14b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9d0c06f400 0x7f9d0c071910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d2c0cc930 0x7f9d2c1f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 --2- 172.21.15.81:0/3896396222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d2c0c6de0 0x7f9d2c14e960 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 >> 172.21.15.81:0/3896396222 conn(0x7f9d2c068ab0 msgr2=0x7f9d2c14c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 shutdown_connections 2024-05-27T06:55:41.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:41.392+0000 7f9d33400640 1 -- 172.21.15.81:0/3896396222 wait complete. 2024-05-27T06:55:41.415 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:43.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:43.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.628+0000 7fb623e00640 1 Processor -- start 2024-05-27T06:55:43.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.628+0000 7fb623e00640 1 -- start start 2024-05-27T06:55:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb62415e6b0 0x7fb6240cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6240c93a0 con 0x7fb6240cebf0 2024-05-27T06:55:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6240c9660 con 0x7fb62415b050 2024-05-27T06:55:43.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb6240c9170 con 0x7fb62415e6b0 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb622000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb6240cebf0 0x7fb62415dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43340/0 (socket says 172.21.15.81:43340) 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb622000640 1 -- 172.21.15.81:0/2411365183 learned_addr learned my addr 172.21.15.81:0/2411365183 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb622a00640 1 --1- 172.21.15.81:0/2411365183 >> v1:172.21.15.106:6789/0 conn(0x7fb62415b050 0x7fb624154550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:46862/0 (socket says 172.21.15.81:46862) 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623400640 1 --2- 172.21.15.81:0/2411365183 >> v2:172.21.15.106:3300/0 conn(0x7fb62415e6b0 0x7fb6240cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399509347 0 0) 0x7fb6240c9660 con 0x7fb62415b050 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb608003530 con 0x7fb62415b050 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2879943398 0 0) 0x7fb6240c93a0 con 0x7fb6240cebf0 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6240c9660 con 0x7fb6240cebf0 2024-05-27T06:55:43.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623400640 1 -- 172.21.15.81:0/2411365183 >> v1:172.21.15.106:6789/0 conn(0x7fb62415b050 legacy=0x7fb624154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623400640 1 -- 172.21.15.81:0/2411365183 >> v1:172.21.15.81:6789/0 conn(0x7fb6240cebf0 legacy=0x7fb62415dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.632+0000 7fb623400640 1 -- 172.21.15.81:0/2411365183 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb624074e20 con 0x7fb62415e6b0 2024-05-27T06:55:43.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623400640 1 --2- 172.21.15.81:0/2411365183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb62415e6b0 0x7fb6240cf000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009dc0 tx=0x7fb61c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab54cccd6b5318d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:43.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0b2070 con 0x7fb62415e6b0 2024-05-27T06:55:43.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb61c0a5810 con 0x7fb62415e6b0 2024-05-27T06:55:43.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb621600640 1 -- 172.21.15.81:0/2411365183 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0a5be0 con 0x7fb62415e6b0 2024-05-27T06:55:43.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 -- 172.21.15.81:0/2411365183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb62415e6b0 msgr2=0x7fb6240cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 --2- 172.21.15.81:0/2411365183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb62415e6b0 0x7fb6240cf000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009dc0 tx=0x7fb61c01c950 comp rx=0 tx=0).stop 2024-05-27T06:55:43.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 -- 172.21.15.81:0/2411365183 shutdown_connections 2024-05-27T06:55:43.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 --2- 172.21.15.81:0/2411365183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb62415e6b0 0x7fb6240cf000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 -- 172.21.15.81:0/2411365183 >> 172.21.15.81:0/2411365183 conn(0x7fb624071390 msgr2=0x7fb6240717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:43.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 -- 172.21.15.81:0/2411365183 shutdown_connections 2024-05-27T06:55:43.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.636+0000 7fb623e00640 1 -- 172.21.15.81:0/2411365183 wait complete. 2024-05-27T06:55:43.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.640+0000 7fb623e00640 1 Processor -- start 2024-05-27T06:55:43.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.640+0000 7fb623e00640 1 -- start start 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6240cebf0 0x7fb6241fac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 0x7fb624159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 0x7fb624143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb624168780 con 0x7fb62415b050 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb624168900 con 0x7fb6240cebf0 2024-05-27T06:55:43.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb624168a80 con 0x7fb62415e6b0 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb622000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 0x7fb624159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb622000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 0x7fb624159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56902/0 (socket says 172.21.15.81:56902) 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb622000640 1 -- 172.21.15.81:0/3166565665 learned_addr learned my addr 172.21.15.81:0/3166565665 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 0x7fb624143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb622a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6240cebf0 0x7fb6241fac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 -- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6240cebf0 msgr2=0x7fb6241fac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6240cebf0 0x7fb6241fac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 -- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 msgr2=0x7fb624159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 0x7fb624159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 -- 172.21.15.81:0/3166565665 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6241fb170 con 0x7fb62415e6b0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623400640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 0x7fb624143040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0ac000 tx=0x7fb61c0196f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0be020 con 0x7fb62415e6b0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb61c0b8470 con 0x7fb62415e6b0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb624149c20 con 0x7fb62415e6b0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0b2040 con 0x7fb62415e6b0 2024-05-27T06:55:43.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.644+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6241fa430 con 0x7fb62415e6b0 2024-05-27T06:55:43.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.648+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb62414a140 con 0x7fb62415e6b0 2024-05-27T06:55:43.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.648+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb61c0c1070 con 0x7fb62415e6b0 2024-05-27T06:55:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.648+0000 7fb61aa00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 0x7fb604071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:43.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.648+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fb61c129aa0 con 0x7fb62415e6b0 2024-05-27T06:55:43.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.652+0000 7fb622a00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 0x7fb604071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:43.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.652+0000 7fb622a00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 0x7fb604071910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb614007f60 tx=0x7fb614007ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:43.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.652+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb62414a140 con 0x7fb62415e6b0 2024-05-27T06:55:43.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.972+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6241fa220 con 0x7fb62415e6b0 2024-05-27T06:55:43.969 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:43.969 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:43.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.976+0000 7fb61aa00640 1 -- 172.21.15.81:0/3166565665 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7fb61c016540 con 0x7fb62415e6b0 2024-05-27T06:55:43.970 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.976+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 msgr2=0x7fb604071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.976+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 0x7fb604071910 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb614007f60 tx=0x7fb614007ef0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.976+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 msgr2=0x7fb624143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.976+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 0x7fb624143040 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb61c0ac000 tx=0x7fb61c0196f0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 shutdown_connections 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb62415e6b0 0x7fb624143040 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60406f400 0x7fb604071910 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62415b050 0x7fb624159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 --2- 172.21.15.81:0/3166565665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6240cebf0 0x7fb6241fac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 >> 172.21.15.81:0/3166565665 conn(0x7fb624071390 msgr2=0x7fb624152fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 shutdown_connections 2024-05-27T06:55:43.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:43.980+0000 7fb623e00640 1 -- 172.21.15.81:0/3166565665 wait complete. 2024-05-27T06:55:44.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:46.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:46.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.200+0000 7f91fe800640 1 Processor -- start 2024-05-27T06:55:46.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.200+0000 7f91fe800640 1 -- start start 2024-05-27T06:55:46.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fe800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f91f80ce040 0x7f91f80cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fe800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f91f80cead0 0x7f91f80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fe800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f815d340 con 0x7f91f8156a70 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fe800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f91f80cd250 con 0x7f91f80ce040 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fe800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f91f80cf420 con 0x7f91f80cead0 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f91f80ce040 0x7f91f80cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f91f80ce040 0x7f91f80cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56704/0 (socket says 172.21.15.81:56704) 2024-05-27T06:55:46.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 -- 172.21.15.81:0/262654100 learned_addr learned my addr 172.21.15.81:0/262654100 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:46.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fde00640 1 --2- 172.21.15.81:0/262654100 >> v2:172.21.15.106:3300/0 conn(0x7f91f80cead0 0x7f91f80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 -- 172.21.15.81:0/262654100 >> v2:172.21.15.106:3300/0 conn(0x7f91f80cead0 msgr2=0x7f91f80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:46.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 --2- 172.21.15.81:0/262654100 >> v2:172.21.15.106:3300/0 conn(0x7f91f80cead0 0x7f91f80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 -- 172.21.15.81:0/262654100 >> v1:172.21.15.106:6789/0 conn(0x7f91f8156a70 legacy=0x7f91f80d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.204+0000 7f91fd400640 1 -- 172.21.15.81:0/262654100 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f80cd4a0 con 0x7f91f80ce040 2024-05-27T06:55:46.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91fd400640 1 --2- 172.21.15.81:0/262654100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f80cd780 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f91f806e2f0 tx=0x7f91ec01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e6d101e5adc0ee server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:46.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91f7e00640 1 -- 172.21.15.81:0/262654100 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0b0070 con 0x7f91f80ce040 2024-05-27T06:55:46.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91f7e00640 1 -- 172.21.15.81:0/262654100 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec0ab600 con 0x7f91f80ce040 2024-05-27T06:55:46.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91f7e00640 1 -- 172.21.15.81:0/262654100 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0ab9d0 con 0x7f91f80ce040 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91fe800640 1 -- 172.21.15.81:0/262654100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 msgr2=0x7f91f80cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.208+0000 7f91fe800640 1 --2- 172.21.15.81:0/262654100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f80cd780 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f91f806e2f0 tx=0x7f91ec01c490 comp rx=0 tx=0).stop 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 -- 172.21.15.81:0/262654100 shutdown_connections 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 --2- 172.21.15.81:0/262654100 >> v2:172.21.15.106:3300/0 conn(0x7f91f80cead0 0x7f91f80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 --2- 172.21.15.81:0/262654100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f80cd780 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 -- 172.21.15.81:0/262654100 >> 172.21.15.81:0/262654100 conn(0x7f91f806e7c0 msgr2=0x7f91f806ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:46.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 -- 172.21.15.81:0/262654100 shutdown_connections 2024-05-27T06:55:46.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 -- 172.21.15.81:0/262654100 wait complete. 2024-05-27T06:55:46.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 Processor -- start 2024-05-27T06:55:46.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.212+0000 7f91fe800640 1 -- start start 2024-05-27T06:55:46.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91f80cead0 0x7f91f814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91f8156a70 0x7f91f80cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91f80d2730 con 0x7f91f80ce040 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f91f80d25b0 con 0x7f91f8156a70 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f91f81697a0 con 0x7f91f80cead0 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56718/0 (socket says 172.21.15.81:56718) 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 -- 172.21.15.81:0/644874860 learned_addr learned my addr 172.21.15.81:0/644874860 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fca00640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91f80cead0 0x7f91f814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fde00640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91f8156a70 0x7f91f80cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 -- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91f80cead0 msgr2=0x7f91f814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91f80cead0 0x7f91f814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 -- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91f8156a70 msgr2=0x7f91f80cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91f8156a70 0x7f91f80cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 -- 172.21.15.81:0/644874860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f8152780 con 0x7f91f80ce040 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fd400640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0ae040 tx=0x7f91ec019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0b0040 con 0x7f91f80ce040 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec0b4640 con 0x7f91f80ce040 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f815fb40 con 0x7f91f80ce040 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0b4a30 con 0x7f91f80ce040 2024-05-27T06:55:46.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.216+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f8151480 con 0x7f91f80ce040 2024-05-27T06:55:46.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.220+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f8050870 con 0x7f91f80ce040 2024-05-27T06:55:46.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.220+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f91ec0ab4c0 con 0x7f91f80ce040 2024-05-27T06:55:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.220+0000 7f91f5600640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 0x7f91d8071960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.220+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f91ec128dd0 con 0x7f91f80ce040 2024-05-27T06:55:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.220+0000 7f91fca00640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 0x7f91d8071960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.224+0000 7f91fca00640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 0x7f91d8071960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007fd0 tx=0x7f91e4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:46.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.228+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f91ec016110 con 0x7f91f80ce040 2024-05-27T06:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.480+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f80d1820 con 0x7f91f80ce040 2024-05-27T06:55:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.480+0000 7f91f5600640 1 -- 172.21.15.81:0/644874860 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7f91ec0f5650 con 0x7f91f80ce040 2024-05-27T06:55:46.475 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:46.475 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 msgr2=0x7f91d8071960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 0x7f91d8071960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f91e4007fd0 tx=0x7f91e4009040 comp rx=0 tx=0).stop 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 msgr2=0x7f91f8152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0ae040 tx=0x7f91ec019230 comp rx=0 tx=0).stop 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 shutdown_connections 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f91f80cead0 0x7f91f814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f91d806f450 0x7f91d8071960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f91f8156a70 0x7f91f80cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 --2- 172.21.15.81:0/644874860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91f80ce040 0x7f91f8152240 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:46.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 >> 172.21.15.81:0/644874860 conn(0x7f91f806e7c0 msgr2=0x7f91f80c85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:46.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 shutdown_connections 2024-05-27T06:55:46.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:46.484+0000 7f91fe800640 1 -- 172.21.15.81:0/644874860 wait complete. 2024-05-27T06:55:46.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:48.497 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:48.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 Processor -- start 2024-05-27T06:55:48.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 -- start start 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f272c0c6de0 0x7f272c0cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f272c0cc930 0x7f272c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272c0755a0 con 0x7f272c15a3e0 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f272c0ca9f0 con 0x7f272c0cc930 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2734600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f272c0c7400 con 0x7f272c0c6de0 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f272c0cc930 0x7f272c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f272c0cc930 0x7f272c0ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56730/0 (socket says 172.21.15.81:56730) 2024-05-27T06:55:48.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 -- 172.21.15.81:0/2777657020 learned_addr learned my addr 172.21.15.81:0/2777657020 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2731200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f272c15a3e0 0x7f272c0ca2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59124/0 (socket says 172.21.15.81:59124) 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2731c00640 1 --2- 172.21.15.81:0/2777657020 >> v2:172.21.15.106:3300/0 conn(0x7f272c0c6de0 0x7f272c0cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 -- 172.21.15.81:0/2777657020 >> v2:172.21.15.106:3300/0 conn(0x7f272c0c6de0 msgr2=0x7f272c0cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 --2- 172.21.15.81:0/2777657020 >> v2:172.21.15.106:3300/0 conn(0x7f272c0c6de0 0x7f272c0cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 -- 172.21.15.81:0/2777657020 >> v1:172.21.15.81:6789/0 conn(0x7f272c15a3e0 legacy=0x7f272c0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f2732600640 1 -- 172.21.15.81:0/2777657020 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c0c98c0 con 0x7f272c0cc930 2024-05-27T06:55:48.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.656+0000 7f272be00640 1 -- 172.21.15.81:0/2777657020 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340362688 0 0) 0x7f272c0755a0 con 0x7f272c15a3e0 2024-05-27T06:55:48.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2732600640 1 --2- 172.21.15.81:0/2777657020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c0cc930 0x7f272c0ccd40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f272c0d0930 tx=0x7f272401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5627e0e04f9b415e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:48.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f272be00640 1 -- 172.21.15.81:0/2777657020 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27240b4070 con 0x7f272c0cc930 2024-05-27T06:55:48.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f272be00640 1 -- 172.21.15.81:0/2777657020 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27240af400 con 0x7f272c0cc930 2024-05-27T06:55:48.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f272be00640 1 -- 172.21.15.81:0/2777657020 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27240af7d0 con 0x7f272c0cc930 2024-05-27T06:55:48.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 -- 172.21.15.81:0/2777657020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c0cc930 msgr2=0x7f272c0ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 --2- 172.21.15.81:0/2777657020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c0cc930 0x7f272c0ccd40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f272c0d0930 tx=0x7f272401f1e0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 -- 172.21.15.81:0/2777657020 shutdown_connections 2024-05-27T06:55:48.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 --2- 172.21.15.81:0/2777657020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c0cc930 0x7f272c0ccd40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 --2- 172.21.15.81:0/2777657020 >> v2:172.21.15.106:3300/0 conn(0x7f272c0c6de0 0x7f272c0cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 -- 172.21.15.81:0/2777657020 >> 172.21.15.81:0/2777657020 conn(0x7f272c068ab0 msgr2=0x7f272c068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:48.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 -- 172.21.15.81:0/2777657020 shutdown_connections 2024-05-27T06:55:48.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.660+0000 7f2734600640 1 -- 172.21.15.81:0/2777657020 wait complete. 2024-05-27T06:55:48.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 Processor -- start 2024-05-27T06:55:48.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 -- start start 2024-05-27T06:55:48.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 0x7f272c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 0x7f272c151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f272c15a7f0 con 0x7f272c15a3e0 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f272c0cb820 con 0x7f272c0cc930 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2734600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f272c0c95c0 con 0x7f272c0c6de0 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56738/0 (socket says 172.21.15.81:56738) 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 -- 172.21.15.81:0/3061488679 learned_addr learned my addr 172.21.15.81:0/3061488679 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2731200640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 0x7f272c151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2731c00640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 0x7f272c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 -- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 msgr2=0x7f272c14c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 0x7f272c14c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 -- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 msgr2=0x7f272c151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 0x7f272c151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2732600640 1 -- 172.21.15.81:0/3061488679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c14cb60 con 0x7f272c15a3e0 2024-05-27T06:55:48.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2731200640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 0x7f272c151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:55:48.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.664+0000 7f2731c00640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 0x7f272c14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2732600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f27240b2040 tx=0x7f272401bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27240b4040 con 0x7f272c15a3e0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27240090d0 con 0x7f272c15a3e0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272c14d4d0 con 0x7f272c15a3e0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724009510 con 0x7f272c15a3e0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f272c14da90 con 0x7f272c15a3e0 2024-05-27T06:55:48.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f272c14e150 con 0x7f272c15a3e0 2024-05-27T06:55:48.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2724003be0 con 0x7f272c15a3e0 2024-05-27T06:55:48.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 0x7f270c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:48.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.668+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7f272412d1d0 con 0x7f272c15a3e0 2024-05-27T06:55:48.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.672+0000 7f2731c00640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 0x7f270c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:48.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.672+0000 7f2731c00640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 0x7f270c071910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2720009b10 tx=0x7f2720008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:48.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.672+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f272c14e150 con 0x7f272c15a3e0 2024-05-27T06:55:48.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.948+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f272c06a560 con 0x7f272c15a3e0 2024-05-27T06:55:48.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2729600640 1 -- 172.21.15.81:0/3061488679 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7f2724018d40 con 0x7f272c15a3e0 2024-05-27T06:55:48.944 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:48.945 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:48.945 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:48.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 msgr2=0x7f270c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 0x7f270c071910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2720009b10 tx=0x7f2720008040 comp rx=0 tx=0).stop 2024-05-27T06:55:48.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 msgr2=0x7f272c14a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:48.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f27240b2040 tx=0x7f272401bf80 comp rx=0 tx=0).stop 2024-05-27T06:55:48.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 shutdown_connections 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f270c06f400 0x7f270c071910 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272c15a3e0 0x7f272c14a500 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f272c0cc930 0x7f272c151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 --2- 172.21.15.81:0/3061488679 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f272c0c6de0 0x7f272c14c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 >> 172.21.15.81:0/3061488679 conn(0x7f272c068ab0 msgr2=0x7f272c0749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 shutdown_connections 2024-05-27T06:55:48.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:48.952+0000 7f2734600640 1 -- 172.21.15.81:0/3061488679 wait complete. 2024-05-27T06:55:48.971 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:50.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 Processor -- start 2024-05-27T06:55:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 -- start start 2024-05-27T06:55:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fce60156a60 0x7fce600c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fce600744f0 0x7fce60074900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce60160b10 con 0x7fce60160700 2024-05-27T06:55:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fce6006d5a0 con 0x7fce60156a60 2024-05-27T06:55:51.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce65800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fce6006d880 con 0x7fce600744f0 2024-05-27T06:55:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce5ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fce60156a60 0x7fce600c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce5ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fce60156a60 0x7fce600c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56750/0 (socket says 172.21.15.81:56750) 2024-05-27T06:55:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce5ea00640 1 -- 172.21.15.81:0/1912295245 learned_addr learned my addr 172.21.15.81:0/1912295245 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:51.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.136+0000 7fce5f400640 1 --2- 172.21.15.81:0/1912295245 >> v2:172.21.15.106:3300/0 conn(0x7fce600744f0 0x7fce60074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.140+0000 7fce5ea00640 1 -- 172.21.15.81:0/1912295245 >> v2:172.21.15.106:3300/0 conn(0x7fce600744f0 msgr2=0x7fce60074900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.140+0000 7fce5ea00640 1 --2- 172.21.15.81:0/1912295245 >> v2:172.21.15.106:3300/0 conn(0x7fce600744f0 0x7fce60074900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.140+0000 7fce5ea00640 1 -- 172.21.15.81:0/1912295245 >> v1:172.21.15.81:6789/0 conn(0x7fce60160700 legacy=0x7fce60074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.140+0000 7fce5ea00640 1 -- 172.21.15.81:0/1912295245 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6006db60 con 0x7fce60156a60 2024-05-27T06:55:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.140+0000 7fce5ea00640 1 --2- 172.21.15.81:0/1912295245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60156a60 0x7fce600c92d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce6006c890 tx=0x7fce5001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd958ef5958b130 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce5d600640 1 -- 172.21.15.81:0/1912295245 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce500b0070 con 0x7fce60156a60 2024-05-27T06:55:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce5d600640 1 -- 172.21.15.81:0/1912295245 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce500ab600 con 0x7fce60156a60 2024-05-27T06:55:51.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce5d600640 1 -- 172.21.15.81:0/1912295245 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce500ab9d0 con 0x7fce60156a60 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 -- 172.21.15.81:0/1912295245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60156a60 msgr2=0x7fce600c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 --2- 172.21.15.81:0/1912295245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60156a60 0x7fce600c92d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fce6006c890 tx=0x7fce5001c520 comp rx=0 tx=0).stop 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 -- 172.21.15.81:0/1912295245 shutdown_connections 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 --2- 172.21.15.81:0/1912295245 >> v2:172.21.15.106:3300/0 conn(0x7fce600744f0 0x7fce60074900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 --2- 172.21.15.81:0/1912295245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60156a60 0x7fce600c92d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 -- 172.21.15.81:0/1912295245 >> 172.21.15.81:0/1912295245 conn(0x7fce60071770 msgr2=0x7fce600ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 -- 172.21.15.81:0/1912295245 shutdown_connections 2024-05-27T06:55:51.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.144+0000 7fce65800640 1 -- 172.21.15.81:0/1912295245 wait complete. 2024-05-27T06:55:51.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 Processor -- start 2024-05-27T06:55:51.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 -- start start 2024-05-27T06:55:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce600744f0 0x7fce601f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce60156a60 0x7fce6014c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fce600c8d90 con 0x7fce60160700 2024-05-27T06:55:51.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fce600c8c10 con 0x7fce60156a60 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce65800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fce601697d0 con 0x7fce600744f0 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56760/0 (socket says 172.21.15.81:56760) 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 -- 172.21.15.81:0/947078410 learned_addr learned my addr 172.21.15.81:0/947078410 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5ea00640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce600744f0 0x7fce601f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5e000640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce60156a60 0x7fce6014c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 -- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce600744f0 msgr2=0x7fce601f99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce600744f0 0x7fce601f99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 -- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce60156a60 msgr2=0x7fce6014c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce60156a60 0x7fce6014c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.148+0000 7fce5f400640 1 -- 172.21.15.81:0/947078410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce600ca530 con 0x7fce60160700 2024-05-27T06:55:51.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce5f400640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ef90 tx=0x7fce5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58018020 con 0x7fce60160700 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce58014480 con 0x7fce60160700 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58009e60 con 0x7fce60160700 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce601f9f20 con 0x7fce60160700 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce600c9c90 con 0x7fce60160700 2024-05-27T06:55:51.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.152+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce6006a560 con 0x7fce60160700 2024-05-27T06:55:51.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.156+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fce5801b020 con 0x7fce60160700 2024-05-27T06:55:51.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.156+0000 7fce56a00640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 0x7fce40071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:51.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.156+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6255+0+0 (secure 0 0 0) 0x7fce58090080 con 0x7fce60160700 2024-05-27T06:55:51.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.156+0000 7fce5ea00640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 0x7fce40071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:51.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.156+0000 7fce5ea00640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 0x7fce40071940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fce500ad000 tx=0x7fce50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:51.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.164+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fce58022030 con 0x7fce60160700 2024-05-27T06:55:51.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.412+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce600c8850 con 0x7fce60160700 2024-05-27T06:55:51.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce56a00640 1 -- 172.21.15.81:0/947078410 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6915 (secure 0 0 0) 0x7fce58010040 con 0x7fce60160700 2024-05-27T06:55:51.408 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:51.409 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5448,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:54:18.746720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4302,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":5,"addr":"172.21.15.106:6836/730205609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":730205609},{"type":"v1","addr":"172.21.15.106:6836","nonce":730205609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4302,"qdb_cluster":[5184,5049,4949,4870,4302]},"id":1}]} 2024-05-27T06:55:51.409 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 47 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 msgr2=0x7fce40071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 0x7fce40071940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fce500ad000 tx=0x7fce50008040 comp rx=0 tx=0).stop 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 msgr2=0x7fce600cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce5800ef90 tx=0x7fce5800c490 comp rx=0 tx=0).stop 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 shutdown_connections 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fce4006f430 0x7fce40071940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce60160700 0x7fce600cc7d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fce60156a60 0x7fce6014c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 --2- 172.21.15.81:0/947078410 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fce600744f0 0x7fce601f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:51.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 >> 172.21.15.81:0/947078410 conn(0x7fce60071770 msgr2=0x7fce600cbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:51.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 shutdown_connections 2024-05-27T06:55:51.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:51.416+0000 7fce65800640 1 -- 172.21.15.81:0/947078410 wait complete. 2024-05-27T06:55:51.434 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-05-27T06:55:53.435 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:53.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 Processor -- start 2024-05-27T06:55:53.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 -- start start 2024-05-27T06:55:53.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 0x7f43040735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 0x7f430406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4304154c00 con 0x7f430406c790 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f43040755b0 con 0x7f43041547f0 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4309600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f43040752d0 con 0x7f43040c9f70 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4302000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f430406c790 0x7f43040c9860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59154/0 (socket says 172.21.15.81:59154) 2024-05-27T06:55:53.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4302000640 1 -- 172.21.15.81:0/2631225000 learned_addr learned my addr 172.21.15.81:0/2631225000 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4302a00640 1 --2- 172.21.15.81:0/2631225000 >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 0x7f43040735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4303400640 1 --2- 172.21.15.81:0/2631225000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 0x7f430406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4303400640 1 -- 172.21.15.81:0/2631225000 >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 msgr2=0x7f43040735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4303400640 1 --2- 172.21.15.81:0/2631225000 >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 0x7f43040735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4303400640 1 -- 172.21.15.81:0/2631225000 >> v1:172.21.15.81:6789/0 conn(0x7f430406c790 legacy=0x7f43040c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.624+0000 7f4303400640 1 -- 172.21.15.81:0/2631225000 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43040caaa0 con 0x7f43040c9f70 2024-05-27T06:55:53.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.628+0000 7f4302a00640 1 --2- 172.21.15.81:0/2631225000 >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 0x7f43040735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:55:53.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4303400640 1 --2- 172.21.15.81:0/2631225000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 0x7f430406cba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009d90 tx=0x7f42fc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3d8d46c717f649 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:53.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4301600640 1 -- 172.21.15.81:0/2631225000 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc0b2070 con 0x7f43040c9f70 2024-05-27T06:55:53.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4301600640 1 -- 172.21.15.81:0/2631225000 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42fc0a5830 con 0x7f43040c9f70 2024-05-27T06:55:53.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4301600640 1 -- 172.21.15.81:0/2631225000 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc0a5c00 con 0x7f43040c9f70 2024-05-27T06:55:53.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 -- 172.21.15.81:0/2631225000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 msgr2=0x7f430406cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 --2- 172.21.15.81:0/2631225000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 0x7f430406cba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009d90 tx=0x7f42fc01c970 comp rx=0 tx=0).stop 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 -- 172.21.15.81:0/2631225000 shutdown_connections 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 --2- 172.21.15.81:0/2631225000 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43040c9f70 0x7f430406cba0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 --2- 172.21.15.81:0/2631225000 >> v2:172.21.15.106:3300/0 conn(0x7f43041547f0 0x7f43040735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 -- 172.21.15.81:0/2631225000 >> 172.21.15.81:0/2631225000 conn(0x7f430406dca0 msgr2=0x7f430406e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 -- 172.21.15.81:0/2631225000 shutdown_connections 2024-05-27T06:55:53.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.632+0000 7f4309600640 1 -- 172.21.15.81:0/2631225000 wait complete. 2024-05-27T06:55:53.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 Processor -- start 2024-05-27T06:55:53.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 -- start start 2024-05-27T06:55:53.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 0x7f4304148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43041547f0 0x7f430414f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f430406d560 con 0x7f43040c9f70 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f43040c8090 con 0x7f430406c790 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4309600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4304074f70 con 0x7f43041547f0 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 0x7f4304148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56776/0 (socket says 172.21.15.81:56776) 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302000640 1 -- 172.21.15.81:0/2870268838 learned_addr learned my addr 172.21.15.81:0/2870268838 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:53.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4303400640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43041547f0 0x7f430414f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302a00640 1 -- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43041547f0 msgr2=0x7f430414f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302a00640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43041547f0 0x7f430414f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302a00640 1 -- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 msgr2=0x7f4304149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.636+0000 7f4302a00640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4302a00640 1 -- 172.21.15.81:0/2870268838 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4304149f50 con 0x7f430406c790 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4302000640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4302a00640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 0x7f4304148430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e910 tx=0x7f42f400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f4017070 con 0x7f430406c790 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f400c9a0 con 0x7f430406c790 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f400cd90 con 0x7f430406c790 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43040cd660 con 0x7f430406c790 2024-05-27T06:55:53.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43040cdc00 con 0x7f430406c790 2024-05-27T06:55:53.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f430406a560 con 0x7f430406c790 2024-05-27T06:55:53.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.640+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f42f401a070 con 0x7f430406c790 2024-05-27T06:55:53.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.644+0000 7f42faa00640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 0x7f42e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:53.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.644+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f42f408f560 con 0x7f430406c790 2024-05-27T06:55:53.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.644+0000 7f4302000640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 0x7f42e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:53.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.644+0000 7f4302000640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 0x7f42e4071940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f430406e6c0 tx=0x7f42f0007f10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:53.642 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.648+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f42f405bc00 con 0x7f430406c790 2024-05-27T06:55:53.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.844+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43040c5b20 con 0x7f430406c790 2024-05-27T06:55:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.844+0000 7f42faa00640 1 -- 172.21.15.81:0/2870268838 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f42f405ff00 con 0x7f430406c790 2024-05-27T06:55:53.840 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:53.840 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:55:53.840 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:55:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 msgr2=0x7f42e4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 0x7f42e4071940 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f430406e6c0 tx=0x7f42f0007f10 comp rx=0 tx=0).stop 2024-05-27T06:55:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 msgr2=0x7f4304148430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:53.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 0x7f4304148430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f42f400e910 tx=0x7f42f400edd0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 shutdown_connections 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f43041547f0 0x7f430414f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42e406f430 0x7f42e4071940 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43040c9f70 0x7f4304149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 --2- 172.21.15.81:0/2870268838 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f430406c790 0x7f4304148430 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 >> 172.21.15.81:0/2870268838 conn(0x7f430406dca0 msgr2=0x7f4304071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 shutdown_connections 2024-05-27T06:55:53.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:53.848+0000 7f4309600640 1 -- 172.21.15.81:0/2870268838 wait complete. 2024-05-27T06:55:53.862 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-05-27T06:55:53.862 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-05-27T06:55:53.862 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 Processor -- start 2024-05-27T06:55:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 -- start start 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe604154a10 0x7fe60415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6040caf10 0x7fe6040c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe6040ca250 0x7fe6040ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe6040cfe60 con 0x7fe6040ca250 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fe60415b5e0 con 0x7fe6040caf10 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe609000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe60415a5e0 con 0x7fe604154a10 2024-05-27T06:55:54.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe6040ca250 0x7fe6040ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe6040ca250 0x7fe6040ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56778/0 (socket says 172.21.15.81:56778) 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe601400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fe6040caf10 0x7fe6040c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 -- 172.21.15.81:0/2270361214 learned_addr learned my addr 172.21.15.81:0/2270361214 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe601e00640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe604154a10 0x7fe60415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 -- 172.21.15.81:0/2270361214 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe604154a10 msgr2=0x7fe60415b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:54.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe604154a10 0x7fe60415b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 -- 172.21.15.81:0/2270361214 >> v2:172.21.15.106:3300/0 conn(0x7fe6040caf10 msgr2=0x7fe6040c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:54.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 --2- 172.21.15.81:0/2270361214 >> v2:172.21.15.106:3300/0 conn(0x7fe6040caf10 0x7fe6040c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.024+0000 7fe602800640 1 -- 172.21.15.81:0/2270361214 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6040d00b0 con 0x7fe6040ca250 2024-05-27T06:55:54.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe602800640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6040ca250 0x7fe6040ca660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe604069650 tx=0x7fe5fc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65d9a0e3ce70e02 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:54.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe600a00640 1 -- 172.21.15.81:0/2270361214 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0a5670 con 0x7fe6040ca250 2024-05-27T06:55:54.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe600a00640 1 -- 172.21.15.81:0/2270361214 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5fc0a5810 con 0x7fe6040ca250 2024-05-27T06:55:54.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe600a00640 1 -- 172.21.15.81:0/2270361214 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0a5be0 con 0x7fe6040ca250 2024-05-27T06:55:54.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 -- 172.21.15.81:0/2270361214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6040ca250 msgr2=0x7fe6040ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6040ca250 0x7fe6040ca660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fe604069650 tx=0x7fe5fc01c9f0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 -- 172.21.15.81:0/2270361214 shutdown_connections 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6040ca250 0x7fe6040ca660 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 --2- 172.21.15.81:0/2270361214 >> v2:172.21.15.106:3300/0 conn(0x7fe6040caf10 0x7fe6040c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 --2- 172.21.15.81:0/2270361214 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe604154a10 0x7fe60415b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 -- 172.21.15.81:0/2270361214 >> 172.21.15.81:0/2270361214 conn(0x7fe604068ab0 msgr2=0x7fe604068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 -- 172.21.15.81:0/2270361214 shutdown_connections 2024-05-27T06:55:54.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.032+0000 7fe609000640 1 -- 172.21.15.81:0/2270361214 wait complete. 2024-05-27T06:55:54.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 Processor -- start 2024-05-27T06:55:54.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- start start 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 0x7fe604145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6040caf10 0x7fe60415cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040c9ca0 con 0x7fe604154a10 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe60415ab80 con 0x7fe6040caf10 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe6040c9e20 con 0x7fe6040ca250 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe601e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 0x7fe604145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe601400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6040caf10 0x7fe60415cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56786/0 (socket says 172.21.15.81:56786) 2024-05-27T06:55:54.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 -- 172.21.15.81:0/2345838433 learned_addr learned my addr 172.21.15.81:0/2345838433 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 -- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 msgr2=0x7fe604145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 0x7fe604145920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 -- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6040caf10 msgr2=0x7fe60415cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6040caf10 0x7fe60415cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 -- 172.21.15.81:0/2345838433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe604164cd0 con 0x7fe604154a10 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe601e00640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 0x7fe604145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe602800640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0af040 tx=0x7fe5fc019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc0bc020 con 0x7fe604154a10 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5fc0b8690 con 0x7fe604154a10 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60406f060 con 0x7fe604154a10 2024-05-27T06:55:54.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc009510 con 0x7fe604154a10 2024-05-27T06:55:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.036+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60406f620 con 0x7fe604154a10 2024-05-27T06:55:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe604155150 con 0x7fe604154a10 2024-05-27T06:55:54.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe5fc0bf020 con 0x7fe604154a10 2024-05-27T06:55:54.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe5fa000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 0x7fe5e4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:54.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fe5fc12a510 con 0x7fe604154a10 2024-05-27T06:55:54.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe601e00640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 0x7fe5e4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:54.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.040+0000 7fe601e00640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 0x7fe5e4071910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009eb0 tx=0x7fe5f4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:54.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.044+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe5fc0165e0 con 0x7fe604154a10 2024-05-27T06:55:54.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.240+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe60406a560 con 0x7fe604154a10 2024-05-27T06:55:54.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.240+0000 7fe5fa000640 1 -- 172.21.15.81:0/2345838433 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7fe5fc0f6c70 con 0x7fe604154a10 2024-05-27T06:55:54.235 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:54.235 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:55:54.235 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:55:54.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 msgr2=0x7fe5e4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:54.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 0x7fe5e4071910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009eb0 tx=0x7fe5f4009190 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 msgr2=0x7fe60415d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc0af040 tx=0x7fe5fc019790 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 shutdown_connections 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe5e406f400 0x7fe5e4071910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe604154a10 0x7fe60415d450 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe6040caf10 0x7fe60415cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 --2- 172.21.15.81:0/2345838433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe6040ca250 0x7fe604145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 >> 172.21.15.81:0/2345838433 conn(0x7fe604068ab0 msgr2=0x7fe6040cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 shutdown_connections 2024-05-27T06:55:54.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:54.244+0000 7fe609000640 1 -- 172.21.15.81:0/2345838433 wait complete. 2024-05-27T06:55:54.258 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:55:53.380697+0000", "qdb_cluster": [ 4870, 4949, 5049, 5184 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:55:56.261 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 Processor -- start 2024-05-27T06:55:56.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 -- start start 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f48dc0ca370 0x7f48dc154980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15e090 0x7f48dc164fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48dc15e4a0 con 0x7f48dc15dc80 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f48dc0732b0 con 0x7f48dc0ca370 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48e2e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc072e20 con 0x7f48dc15e090 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48dbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f48dc0ca370 0x7f48dc154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48dbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f48dc0ca370 0x7f48dc154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54040/0 (socket says 172.21.15.81:54040) 2024-05-27T06:55:56.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.432+0000 7f48dbe00640 1 -- 172.21.15.81:0/1482034761 learned_addr learned my addr 172.21.15.81:0/1482034761 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:56.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48e0e00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15e090 0x7f48dc164fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48dbe00640 1 -- 172.21.15.81:0/1482034761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15e090 msgr2=0x7f48dc164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:56.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48dbe00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15e090 0x7f48dc164fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48dbe00640 1 -- 172.21.15.81:0/1482034761 >> v1:172.21.15.106:6789/0 conn(0x7f48dc15dc80 legacy=0x7f48dc0c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48dbe00640 1 -- 172.21.15.81:0/1482034761 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc06c910 con 0x7f48dc0ca370 2024-05-27T06:55:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48dbe00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc0ca370 0x7f48dc154980 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009870 tx=0x7f48cc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7352241e5e71a51 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48daa00640 1 -- 172.21.15.81:0/1482034761 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc0a9640 con 0x7f48dc0ca370 2024-05-27T06:55:56.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48daa00640 1 -- 172.21.15.81:0/1482034761 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48cc0a97e0 con 0x7f48dc0ca370 2024-05-27T06:55:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.436+0000 7f48daa00640 1 -- 172.21.15.81:0/1482034761 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc0a9bb0 con 0x7f48dc0ca370 2024-05-27T06:55:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- 172.21.15.81:0/1482034761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc0ca370 msgr2=0x7f48dc154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc0ca370 0x7f48dc154980 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009870 tx=0x7f48cc01c560 comp rx=0 tx=0).stop 2024-05-27T06:55:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- 172.21.15.81:0/1482034761 shutdown_connections 2024-05-27T06:55:56.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15e090 0x7f48dc164fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 --2- 172.21.15.81:0/1482034761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc0ca370 0x7f48dc154980 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- 172.21.15.81:0/1482034761 >> 172.21.15.81:0/1482034761 conn(0x7f48dc06f230 msgr2=0x7f48dc070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- 172.21.15.81:0/1482034761 shutdown_connections 2024-05-27T06:55:56.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- 172.21.15.81:0/1482034761 wait complete. 2024-05-27T06:55:56.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 Processor -- start 2024-05-27T06:55:56.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.440+0000 7f48e2e00640 1 -- start start 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48dc0ca370 0x7f48dc14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15dc80 0x7f48dc148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc06d1d0 con 0x7f48dc15e090 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc154ec0 con 0x7f48dc0ca370 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc068c70 con 0x7f48dc15dc80 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48dbe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48dc0ca370 0x7f48dc14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54054/0 (socket says 172.21.15.81:54054) 2024-05-27T06:55:56.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 -- 172.21.15.81:0/669193398 learned_addr learned my addr 172.21.15.81:0/669193398 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48db400640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15dc80 0x7f48dc148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 -- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15dc80 msgr2=0x7f48dc148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15dc80 0x7f48dc148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 -- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48dc0ca370 msgr2=0x7f48dc14f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48dc0ca370 0x7f48dc14f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 -- 172.21.15.81:0/669193398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc15cd70 con 0x7f48dc15e090 2024-05-27T06:55:56.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e0e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f48d4007e60 tx=0x7f48d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4014420 con 0x7f48dc15e090 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48d4019070 con 0x7f48dc15e090 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48dc1522e0 con 0x7f48dc15e090 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4014820 con 0x7f48dc15e090 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48dc15c710 con 0x7f48dc15e090 2024-05-27T06:55:56.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48dc06a560 con 0x7f48dc15e090 2024-05-27T06:55:56.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.444+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f48d40149c0 con 0x7f48dc15e090 2024-05-27T06:55:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.448+0000 7f48d3e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 0x7f48bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.448+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f48d408fd10 con 0x7f48dc15e090 2024-05-27T06:55:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.448+0000 7f48dbe00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 0x7f48bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:56.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.448+0000 7f48dbe00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 0x7f48bc071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0ae000 tx=0x7f48cc0b3040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:56.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.452+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f48d4095020 con 0x7f48dc15e090 2024-05-27T06:55:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48dc073a20 con 0x7f48dc15e090 2024-05-27T06:55:56.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48d3e00640 1 -- 172.21.15.81:0/669193398 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f48d405c290 con 0x7f48dc15e090 2024-05-27T06:55:56.633 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:56.633 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:55:56.633 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 msgr2=0x7f48bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 0x7f48bc071910 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f48cc0ae000 tx=0x7f48cc0b3040 comp rx=0 tx=0).stop 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 msgr2=0x7f48dc164300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f48d4007e60 tx=0x7f48d400e4b0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 shutdown_connections 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f48dc15dc80 0x7f48dc148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f48bc06f400 0x7f48bc071910 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48dc15e090 0x7f48dc164300 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 --2- 172.21.15.81:0/669193398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48dc0ca370 0x7f48dc14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 >> 172.21.15.81:0/669193398 conn(0x7f48dc06f230 msgr2=0x7f48dc0caf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 shutdown_connections 2024-05-27T06:55:56.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:56.644+0000 7f48e2e00640 1 -- 172.21.15.81:0/669193398 wait complete. 2024-05-27T06:55:56.654 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:55:58.634 INFO:tasks.ceph.mds.a.smithi081.stderr:./src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4ad7c00640 time 2024-05-27T06:55:58.646410+0000 2024-05-27T06:55:58.634 INFO:tasks.ceph.mds.a.smithi081.stderr:./src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ae8e3baac] 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.635 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-05-27T06:55:58.644+0000 7f4ad7c00640 -1 ./src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4ad7c00640 time 2024-05-27T06:55:58.646410+0000 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr:./src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T06:55:58.636 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ae8e3baac] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.637 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr:*** Caught signal (Aborted) ** 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: in thread 7f4ad7c00640 thread_name:md_log_replay 2024-05-27T06:55:58.638 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ae8442520] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: pthread_kill() 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: raise() 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: abort() 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x262) [0x7f4ae8e3bbe7] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.639 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 17: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr:2024-05-27T06:55:58.648+0000 7f4ad7c00640 -1 *** Caught signal (Aborted) ** 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: in thread 7f4ad7c00640 thread_name:md_log_replay 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ae8442520] 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: pthread_kill() 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: raise() 2024-05-27T06:55:58.640 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: abort() 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x262) [0x7f4ae8e3bbe7] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.641 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: 17: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T06:55:58.642 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.644 INFO:tasks.ceph.mds.a.smithi081.stderr: -1> 2024-05-27T06:55:58.644+0000 7f4ad7c00640 -1 ./src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4ad7c00640 time 2024-05-27T06:55:58.646410+0000 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr:./src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ae8e3baac] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.645 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 0> 2024-05-27T06:55:58.648+0000 7f4ad7c00640 -1 *** Caught signal (Aborted) ** 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: in thread 7f4ad7c00640 thread_name:md_log_replay 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.646 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ae8442520] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: pthread_kill() 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: raise() 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: abort() 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x262) [0x7f4ae8e3bbe7] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.647 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 17: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T06:55:58.648 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.653 INFO:tasks.ceph.mds.a.smithi081.stderr: -1> 2024-05-27T06:55:58.644+0000 7f4ad7c00640 -1 ./src/mds/SnapClient.cc: In function 'std::set SnapClient::filter(const std::set&) const' thread 7f4ad7c00640 time 2024-05-27T06:55:58.646410+0000 2024-05-27T06:55:58.653 INFO:tasks.ceph.mds.a.smithi081.stderr:./src/mds/SnapClient.cc: 197: FAILED ceph_assert(cached_version > 0) 2024-05-27T06:55:58.653 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.653 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.653 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f4ae8e3baac] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.654 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 0> 2024-05-27T06:55:58.648+0000 7f4ad7c00640 -1 *** Caught signal (Aborted) ** 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: in thread 7f4ad7c00640 thread_name:md_log_replay 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: ceph version 19.0.0-3872-g38f9d900 (38f9d900c83c63799fbdbe61acc9a11b0d3554a6) squid (dev) 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f4ae8442520] 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 2: pthread_kill() 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 3: raise() 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 4: abort() 2024-05-27T06:55:58.655 INFO:tasks.ceph.mds.a.smithi081.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x262) [0x7f4ae8e3bbe7] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x23bcb9) [0x7f4ae8e3bcb9] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 7: (SnapClient::filter(std::set, std::allocator > const&) const+0x81) [0x5555e16c1239] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 8: (SnapRealm::build_snap_set() const+0x22f) [0x5555e16b88c7] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 9: (SnapRealm::check_cache() const+0x156) [0x5555e16b8bb2] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 10: (operator<<(std::ostream&, SnapRealm const&)+0x1f2) [0x5555e16b8f78] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 11: (CInode::decode_snap_blob(ceph::buffer::v15_2_0::list const&)+0x1f2) [0x5555e16728b6] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 12: (EMetaBlob::fullbit::update_inode(MDSRank*, CInode*)+0x1c9) [0x5555e13bd351] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 13: (EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*)+0x1c64) [0x5555e13c9ef4] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 14: (EUpdate::replay(MDSRank*)+0x5e) [0x5555e13d7c60] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 15: (MDLog::_replay_thread()+0x1c58) [0x5555e16f0848] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 16: ceph-mds(+0x4a2e63) [0x5555e16dde63] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 17: (Thread::entry_wrapper()+0x43) [0x7f4ae8e1387f] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f4ae8e1389b] 2024-05-27T06:55:58.656 INFO:tasks.ceph.mds.a.smithi081.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f4ae8494b43] 2024-05-27T06:55:58.657 INFO:tasks.ceph.mds.a.smithi081.stderr: 20: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f4ae8526a00] 2024-05-27T06:55:58.657 INFO:tasks.ceph.mds.a.smithi081.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-27T06:55:58.657 INFO:tasks.ceph.mds.a.smithi081.stderr: 2024-05-27T06:55:58.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:55:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 Processor -- start 2024-05-27T06:55:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 -- start start 2024-05-27T06:55:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f88641546f0 0x7f8864072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f88640c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886415db00 con 0x7f88640c7510 2024-05-27T06:55:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f886415e6f0 con 0x7f88641546f0 2024-05-27T06:55:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8868c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f88640755b0 con 0x7f88640732f0 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f88641546f0 0x7f8864072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8862800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f88640c7510 0x7f886415a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:49342/0 (socket says 172.21.15.81:49342) 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f88641546f0 0x7f8864072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54064/0 (socket says 172.21.15.81:54064) 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8862800640 1 -- 172.21.15.81:0/1213650038 learned_addr learned my addr 172.21.15.81:0/1213650038 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861400640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f88640c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 -- 172.21.15.81:0/1213650038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 msgr2=0x7f88640c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:55:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f88640c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 -- 172.21.15.81:0/1213650038 >> v1:172.21.15.81:6789/0 conn(0x7f88640c7510 legacy=0x7f886415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861e00640 1 -- 172.21.15.81:0/1213650038 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8864073860 con 0x7f88641546f0 2024-05-27T06:55:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.820+0000 7f8861400640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f88640c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:55:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8861e00640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864072db0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f88540097c0 tx=0x7f885401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb06742a3d790b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8860a00640 1 -- 172.21.15.81:0/1213650038 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88540b5070 con 0x7f88641546f0 2024-05-27T06:55:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8860a00640 1 -- 172.21.15.81:0/1213650038 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88540a8990 con 0x7f88641546f0 2024-05-27T06:55:58.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8860a00640 1 -- 172.21.15.81:0/1213650038 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88540a8d60 con 0x7f88641546f0 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 -- 172.21.15.81:0/1213650038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 msgr2=0x7f8864072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864072db0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f88540097c0 tx=0x7f885401ec90 comp rx=0 tx=0).stop 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 -- 172.21.15.81:0/1213650038 shutdown_connections 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f88640c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 --2- 172.21.15.81:0/1213650038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864072db0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 -- 172.21.15.81:0/1213650038 >> 172.21.15.81:0/1213650038 conn(0x7f8864070700 msgr2=0x7f88640cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 -- 172.21.15.81:0/1213650038 shutdown_connections 2024-05-27T06:55:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.824+0000 7f8868c00640 1 -- 172.21.15.81:0/1213650038 wait complete. 2024-05-27T06:55:58.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 Processor -- start 2024-05-27T06:55:58.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 -- start start 2024-05-27T06:55:58.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f886414ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88640c7510 0x7f88641482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f886415e380 con 0x7f88641546f0 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f886415e0a0 con 0x7f88640c7510 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8868c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8864154b00 con 0x7f88640732f0 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54074/0 (socket says 172.21.15.81:54074) 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 -- 172.21.15.81:0/894490956 learned_addr learned my addr 172.21.15.81:0/894490956 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8861e00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f886414ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8861400640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88640c7510 0x7f88641482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 -- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 msgr2=0x7f886414ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f886414ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 -- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88640c7510 msgr2=0x7f88641482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88640c7510 0x7f88641482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:58.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 -- 172.21.15.81:0/894490956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88641f3f10 con 0x7f88641546f0 2024-05-27T06:55:58.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.828+0000 7f8862800640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f885c00cb40 tx=0x7f885c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:58.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c017070 con 0x7f88641546f0 2024-05-27T06:55:58.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f885c00ac40 con 0x7f88641546f0 2024-05-27T06:55:58.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88641f3150 con 0x7f88641546f0 2024-05-27T06:55:58.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885c012400 con 0x7f88641546f0 2024-05-27T06:55:58.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88641f3710 con 0x7f88641546f0 2024-05-27T06:55:58.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886415a590 con 0x7f88641546f0 2024-05-27T06:55:58.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f885c01a070 con 0x7f88641546f0 2024-05-27T06:55:58.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 0x7f8844071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:55:58.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f885c08f0c0 con 0x7f88641546f0 2024-05-27T06:55:58.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.832+0000 7f8861e00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 0x7f8844071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:55:58.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.836+0000 7f8861e00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 0x7f8844071940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88540b2000 tx=0x7f8854008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:55:58.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:58.844+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f885c05b770 con 0x7f88641546f0 2024-05-27T06:55:59.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f886406a560 con 0x7f88641546f0 2024-05-27T06:55:59.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f885a000640 1 -- 172.21.15.81:0/894490956 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f885c05fa70 con 0x7f88641546f0 2024-05-27T06:55:59.037 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:55:59.037 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:55:59.038 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:55:59.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 msgr2=0x7f8844071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:59.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 0x7f8844071940 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88540b2000 tx=0x7f8854008040 comp rx=0 tx=0).stop 2024-05-27T06:55:59.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 msgr2=0x7f8864148810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:55:59.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f885c00cb40 tx=0x7f885c00a060 comp rx=0 tx=0).stop 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 shutdown_connections 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f884406f430 0x7f8844071940 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88641546f0 0x7f8864148810 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f88640c7510 0x7f88641482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 --2- 172.21.15.81:0/894490956 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f88640732f0 0x7f886414ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 >> 172.21.15.81:0/894490956 conn(0x7f8864070700 msgr2=0x7f886415b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 shutdown_connections 2024-05-27T06:55:59.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:55:59.048+0000 7f8868c00640 1 -- 172.21.15.81:0/894490956 wait complete. 2024-05-27T06:55:59.058 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:55:59.643 INFO:tasks.ceph.mds.a.smithi081.stderr:daemon-helper: command crashed with signal 6 2024-05-27T06:56:01.059 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 Processor -- start 2024-05-27T06:56:01.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 -- start start 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7facf80c9d20 0x7facf8164e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf806d940 con 0x7facf8153a10 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf806e1c0 con 0x7facf80c5cc0 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facff400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7facf80c7940 con 0x7facf80c9d20 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facf7e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7facf8153a10 0x7facf80c91c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:49354/0 (socket says 172.21.15.81:49354) 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facf7e00640 1 -- 172.21.15.81:0/409955327 learned_addr learned my addr 172.21.15.81:0/409955327 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facfd400640 1 --2- 172.21.15.81:0/409955327 >> v2:172.21.15.81:3300/0 conn(0x7facf80c9d20 0x7facf8164e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facf7400640 1 -- 172.21.15.81:0/409955327 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 924135512 0 0) 0x7facf806d940 con 0x7facf8153a10 2024-05-27T06:56:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facf7400640 1 -- 172.21.15.81:0/409955327 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facdc003530 con 0x7facf8153a10 2024-05-27T06:56:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facfd400640 1 -- 172.21.15.81:0/409955327 >> v1:172.21.15.106:6789/0 conn(0x7facf80c5cc0 legacy=0x7facf815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facfd400640 1 -- 172.21.15.81:0/409955327 >> v1:172.21.15.81:6789/0 conn(0x7facf8153a10 legacy=0x7facf80c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.232+0000 7facfd400640 1 -- 172.21.15.81:0/409955327 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf80c5980 con 0x7facf80c9d20 2024-05-27T06:56:01.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facfd400640 1 --2- 172.21.15.81:0/409955327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf8164e40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7facf80c7ac0 tx=0x7facf001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474664821fc8e349 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facf7400640 1 -- 172.21.15.81:0/409955327 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf00b0070 con 0x7facf80c9d20 2024-05-27T06:56:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facf7400640 1 -- 172.21.15.81:0/409955327 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facf00ab600 con 0x7facf80c9d20 2024-05-27T06:56:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facf7400640 1 -- 172.21.15.81:0/409955327 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf00ab9d0 con 0x7facf80c9d20 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 -- 172.21.15.81:0/409955327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 msgr2=0x7facf8164e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 --2- 172.21.15.81:0/409955327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf8164e40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7facf80c7ac0 tx=0x7facf001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 -- 172.21.15.81:0/409955327 shutdown_connections 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 --2- 172.21.15.81:0/409955327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf8164e40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 -- 172.21.15.81:0/409955327 >> 172.21.15.81:0/409955327 conn(0x7facf80cb2c0 msgr2=0x7facf80cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 -- 172.21.15.81:0/409955327 shutdown_connections 2024-05-27T06:56:01.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.236+0000 7facff400640 1 -- 172.21.15.81:0/409955327 wait complete. 2024-05-27T06:56:01.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 Processor -- start 2024-05-27T06:56:01.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 -- start start 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facf80c5cc0 0x7facf8148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facf8153a10 0x7facf813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7facf80c7f50 con 0x7facf80c9d20 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7facf80c7dd0 con 0x7facf8153a10 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facff400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7facf80c98d0 con 0x7facf80c5cc0 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:01.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54086/0 (socket says 172.21.15.81:54086) 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 -- 172.21.15.81:0/3211401233 learned_addr learned my addr 172.21.15.81:0/3211401233 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facfd400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facf8153a10 0x7facf813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facfca00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facf80c5cc0 0x7facf8148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 -- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facf80c5cc0 msgr2=0x7facf8148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facf80c5cc0 0x7facf8148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 -- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facf8153a10 msgr2=0x7facf813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facf8153a10 0x7facf813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.240+0000 7facf7e00640 1 -- 172.21.15.81:0/3211401233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf8143650 con 0x7facf80c9d20 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf7e00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7face400c430 tx=0x7face400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:01.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4017070 con 0x7facf80c9d20 2024-05-27T06:56:01.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7face4005cf0 con 0x7facf80c9d20 2024-05-27T06:56:01.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face40072d0 con 0x7facf80c9d20 2024-05-27T06:56:01.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf8140430 con 0x7facf80c9d20 2024-05-27T06:56:01.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf8143910 con 0x7facf80c9d20 2024-05-27T06:56:01.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf806a560 con 0x7facf80c9d20 2024-05-27T06:56:01.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7face401a070 con 0x7facf80c9d20 2024-05-27T06:56:01.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 0x7facd8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:01.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7face408e900 con 0x7facf80c9d20 2024-05-27T06:56:01.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.244+0000 7facfca00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 0x7facd8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:01.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.248+0000 7facfca00640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 0x7facd8071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7face80012f0 tx=0x7face8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:01.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.252+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7face405ae80 con 0x7facf80c9d20 2024-05-27T06:56:01.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.440+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf80ce0b0 con 0x7facf80c9d20 2024-05-27T06:56:01.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.440+0000 7facf4c00640 1 -- 172.21.15.81:0/3211401233 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7face405f180 con 0x7facf80c9d20 2024-05-27T06:56:01.431 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:01.431 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:01.431 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:01.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 msgr2=0x7facd8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 0x7facd8071910 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7face80012f0 tx=0x7face8009040 comp rx=0 tx=0).stop 2024-05-27T06:56:01.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 msgr2=0x7facf813f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7face400c430 tx=0x7face400c8f0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 shutdown_connections 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7facd806f400 0x7facd8071910 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7facf8153a10 0x7facf813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facf80c9d20 0x7facf813f9b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 --2- 172.21.15.81:0/3211401233 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7facf80c5cc0 0x7facf8148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 >> 172.21.15.81:0/3211401233 conn(0x7facf80cb2c0 msgr2=0x7facf80ca130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:01.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 shutdown_connections 2024-05-27T06:56:01.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:01.444+0000 7facff400640 1 -- 172.21.15.81:0/3211401233 wait complete. 2024-05-27T06:56:01.453 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:03.454 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:03.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 Processor -- start 2024-05-27T06:56:03.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 -- start start 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 0x7f112c0cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f112c160470 0x7f112c160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c06c790 0x7f112c06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f112c15fd00 con 0x7f112c160470 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f112c15ffe0 con 0x7f112c156900 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1131200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f112c0cd390 con 0x7f112c06c790 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f112c160470 0x7f112c160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f112c160470 0x7f112c160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54098/0 (socket says 172.21.15.81:54098) 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 -- 172.21.15.81:0/1774195036 learned_addr learned my addr 172.21.15.81:0/1774195036 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:03.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1129e00640 1 --2- 172.21.15.81:0/1774195036 >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 0x7f112c0cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f112a800640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c06c790 0x7f112c06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 -- 172.21.15.81:0/1774195036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c06c790 msgr2=0x7f112c06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c06c790 0x7f112c06cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 -- 172.21.15.81:0/1774195036 >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 msgr2=0x7f112c0cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.624+0000 7f1121400640 1 --2- 172.21.15.81:0/1774195036 >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 0x7f112c0cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1121400640 1 -- 172.21.15.81:0/1774195036 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c06f120 con 0x7f112c160470 2024-05-27T06:56:03.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1129e00640 1 --2- 172.21.15.81:0/1774195036 >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 0x7f112c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:03.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1121400640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c160470 0x7f112c160880 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f112c072d90 tx=0x7f111401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623dc4a1017ae68e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:03.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1129400640 1 -- 172.21.15.81:0/1774195036 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11140b5070 con 0x7f112c160470 2024-05-27T06:56:03.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1129400640 1 -- 172.21.15.81:0/1774195036 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11140a89d0 con 0x7f112c160470 2024-05-27T06:56:03.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1129400640 1 -- 172.21.15.81:0/1774195036 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11140a8da0 con 0x7f112c160470 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1131200640 1 -- 172.21.15.81:0/1774195036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c160470 msgr2=0x7f112c160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.628+0000 7f1131200640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c160470 0x7f112c160880 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f112c072d90 tx=0x7f111401ed10 comp rx=0 tx=0).stop 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- 172.21.15.81:0/1774195036 shutdown_connections 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c06c790 0x7f112c06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- 172.21.15.81:0/1774195036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c160470 0x7f112c160880 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- 172.21.15.81:0/1774195036 >> v2:172.21.15.106:3300/0 conn(0x7f112c156900 0x7f112c0cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- 172.21.15.81:0/1774195036 >> 172.21.15.81:0/1774195036 conn(0x7f112c0cb2e0 msgr2=0x7f112c0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:03.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- 172.21.15.81:0/1774195036 shutdown_connections 2024-05-27T06:56:03.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- 172.21.15.81:0/1774195036 wait complete. 2024-05-27T06:56:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 Processor -- start 2024-05-27T06:56:03.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- start start 2024-05-27T06:56:03.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c156900 0x7f112c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f112c160470 0x7f112c15f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f112c06f9c0 con 0x7f112c06c790 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f112c06f6e0 con 0x7f112c160470 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.632+0000 7f1131200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f112c06d5b0 con 0x7f112c156900 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54104/0 (socket says 172.21.15.81:54104) 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1121400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c156900 0x7f112c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 -- 172.21.15.81:0/3505620477 learned_addr learned my addr 172.21.15.81:0/3505620477 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f112a800640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f112c160470 0x7f112c15f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 -- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c156900 msgr2=0x7f112c15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c156900 0x7f112c15d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 -- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f112c160470 msgr2=0x7f112c15f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f112c160470 0x7f112c15f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 -- 172.21.15.81:0/3505620477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c1f2770 con 0x7f112c06c790 2024-05-27T06:56:03.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1129e00640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d830 tx=0x7f111c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:03.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c00b6c0 con 0x7f112c06c790 2024-05-27T06:56:03.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c019070 con 0x7f112c06c790 2024-05-27T06:56:03.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c1f29f0 con 0x7f112c06c790 2024-05-27T06:56:03.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c00baa0 con 0x7f112c06c790 2024-05-27T06:56:03.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c1f2fb0 con 0x7f112c06c790 2024-05-27T06:56:03.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112c06a560 con 0x7f112c06c790 2024-05-27T06:56:03.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.636+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f111c01e460 con 0x7f112c06c790 2024-05-27T06:56:03.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.640+0000 7f1122a00640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 0x7f1108071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:03.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.640+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f111c08fa50 con 0x7f112c06c790 2024-05-27T06:56:03.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.640+0000 7f1121400640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 0x7f1108071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:03.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.640+0000 7f1121400640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 0x7f1108071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f11140b2000 tx=0x7f11140aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:03.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.644+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f112c1f2770 con 0x7f112c06c790 2024-05-27T06:56:03.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.836+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f112c0ccf40 con 0x7f112c06c790 2024-05-27T06:56:03.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.836+0000 7f1122a00640 1 -- 172.21.15.81:0/3505620477 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f111c05c060 con 0x7f112c06c790 2024-05-27T06:56:03.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:03.826 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:03.827 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:03.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 msgr2=0x7f1108071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:03.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 0x7f1108071940 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f11140b2000 tx=0x7f11140aa070 comp rx=0 tx=0).stop 2024-05-27T06:56:03.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 msgr2=0x7f112c15a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d830 tx=0x7f111c00dcf0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 shutdown_connections 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f112c156900 0x7f112c15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f110806f430 0x7f1108071940 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f112c160470 0x7f112c15f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 --2- 172.21.15.81:0/3505620477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c06c790 0x7f112c15a600 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 >> 172.21.15.81:0/3505620477 conn(0x7f112c0cb2e0 msgr2=0x7f112c0c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 shutdown_connections 2024-05-27T06:56:03.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:03.840+0000 7f1131200640 1 -- 172.21.15.81:0/3505620477 wait complete. 2024-05-27T06:56:03.848 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:04.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-05-27T06:56:04.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:56:04.514 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:56:05.849 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:06.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 Processor -- start 2024-05-27T06:56:06.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 -- start start 2024-05-27T06:56:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f267015ac20 0x7f26700c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f267015db00 con 0x7f26700c6fd0 2024-05-27T06:56:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2670073860 con 0x7f26701546f0 2024-05-27T06:56:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f2675000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f267015e6f0 con 0x7f267015ac20 2024-05-27T06:56:06.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266d400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f26700c6fd0 0x7f267015a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45786/0 (socket says 172.21.15.81:45786) 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266d400640 1 -- 172.21.15.81:0/44085236 learned_addr learned my addr 172.21.15.81:0/44085236 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266e800640 1 --2- 172.21.15.81:0/44085236 >> v2:172.21.15.81:3300/0 conn(0x7f267015ac20 0x7f26700c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266ca00640 1 -- 172.21.15.81:0/44085236 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488197664 0 0) 0x7f267015db00 con 0x7f26700c6fd0 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266ca00640 1 -- 172.21.15.81:0/44085236 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2654003530 con 0x7f26700c6fd0 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.020+0000 7f266e800640 1 -- 172.21.15.81:0/44085236 >> v1:172.21.15.106:6789/0 conn(0x7f26701546f0 legacy=0x7f2670072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266e800640 1 -- 172.21.15.81:0/44085236 >> v1:172.21.15.81:6789/0 conn(0x7f26700c6fd0 legacy=0x7f267015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266e800640 1 -- 172.21.15.81:0/44085236 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267015ddc0 con 0x7f267015ac20 2024-05-27T06:56:06.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266e800640 1 --2- 172.21.15.81:0/44085236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f267015ac20 0x7f26700c73e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2668009d90 tx=0x7f266801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8f9cdbe8ca4858 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:06.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266ca00640 1 -- 172.21.15.81:0/44085236 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26680b4070 con 0x7f267015ac20 2024-05-27T06:56:06.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266ca00640 1 -- 172.21.15.81:0/44085236 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26680af600 con 0x7f267015ac20 2024-05-27T06:56:06.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f266ca00640 1 -- 172.21.15.81:0/44085236 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26680af9d0 con 0x7f267015ac20 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f2675000640 1 -- 172.21.15.81:0/44085236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f267015ac20 msgr2=0x7f26700c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f2675000640 1 --2- 172.21.15.81:0/44085236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f267015ac20 0x7f26700c73e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2668009d90 tx=0x7f266801f150 comp rx=0 tx=0).stop 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f2675000640 1 -- 172.21.15.81:0/44085236 shutdown_connections 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f2675000640 1 --2- 172.21.15.81:0/44085236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f267015ac20 0x7f26700c73e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.024+0000 7f2675000640 1 -- 172.21.15.81:0/44085236 >> 172.21.15.81:0/44085236 conn(0x7f2670070700 msgr2=0x7f26700cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- 172.21.15.81:0/44085236 shutdown_connections 2024-05-27T06:56:06.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- 172.21.15.81:0/44085236 wait complete. 2024-05-27T06:56:06.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 Processor -- start 2024-05-27T06:56:06.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- start start 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 0x7f26701e4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267015ac20 0x7f26700ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f267015e380 con 0x7f26700c6fd0 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f26700755b0 con 0x7f267015ac20 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f2675000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2670154b00 con 0x7f26701546f0 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.028+0000 7f266de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44814/0 (socket says 172.21.15.81:44814) 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 -- 172.21.15.81:0/849911455 learned_addr learned my addr 172.21.15.81:0/849911455 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:06.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266e800640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267015ac20 0x7f26700ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266d400640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 0x7f26701e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 -- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 msgr2=0x7f26701e4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 0x7f26701e4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 -- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267015ac20 msgr2=0x7f26700ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267015ac20 0x7f26700ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 -- 172.21.15.81:0/849911455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26701e4c90 con 0x7f26700c6fd0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266d400640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 0x7f26701e4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f266de00640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f265800c3f0 tx=0x7f265800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658017070 con 0x7f26700c6fd0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26580052d0 con 0x7f26700c6fd0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26701e4ea0 con 0x7f26700c6fd0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658003820 con 0x7f26700c6fd0 2024-05-27T06:56:06.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26700c5670 con 0x7f26700c6fd0 2024-05-27T06:56:06.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26701f6080 con 0x7f26700c6fd0 2024-05-27T06:56:06.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.032+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f265801a070 con 0x7f26700c6fd0 2024-05-27T06:56:06.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.036+0000 7f2666000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 0x7f264c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:06.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.036+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f265808e760 con 0x7f26700c6fd0 2024-05-27T06:56:06.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.036+0000 7f266d400640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 0x7f264c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:06.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.036+0000 7f266d400640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 0x7f264c071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f26701e3af0 tx=0x7f2660009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:06.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.040+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2658005480 con 0x7f26700c6fd0 2024-05-27T06:56:06.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f267006a560 con 0x7f26700c6fd0 2024-05-27T06:56:06.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2666000640 1 -- 172.21.15.81:0/849911455 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f265805ae00 con 0x7f26700c6fd0 2024-05-27T06:56:06.221 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:06.221 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 msgr2=0x7f264c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 0x7f264c071910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f26701e3af0 tx=0x7f2660009040 comp rx=0 tx=0).stop 2024-05-27T06:56:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 msgr2=0x7f26700c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f265800c3f0 tx=0x7f265800c8b0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 shutdown_connections 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f26701546f0 0x7f26701e4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f264c06f400 0x7f264c071910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f267015ac20 0x7f26700ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 --2- 172.21.15.81:0/849911455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26700c6fd0 0x7f26700c6a10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 >> 172.21.15.81:0/849911455 conn(0x7f2670070700 msgr2=0x7f26700c9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 shutdown_connections 2024-05-27T06:56:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:06.232+0000 7f2675000640 1 -- 172.21.15.81:0/849911455 wait complete. 2024-05-27T06:56:06.244 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:55:53.380697+0000", "qdb_cluster": [ 4870, 4949, 5049, 5184 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:56:08.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:08.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 Processor -- start 2024-05-27T06:56:08.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 -- start start 2024-05-27T06:56:08.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c072430 0x7fa98c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa98c15a040 0x7fa98c0c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c072840 con 0x7fa98c0c60a0 2024-05-27T06:56:08.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa98c0c5040 con 0x7fa98c15a040 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa993000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c0c75c0 con 0x7fa98c072430 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa98c15a040 0x7fa98c0c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa98c15a040 0x7fa98c0c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44828/0 (socket says 172.21.15.81:44828) 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 -- 172.21.15.81:0/22526070 learned_addr learned my addr 172.21.15.81:0/22526070 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa98be00640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c072430 0x7fa98c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 -- 172.21.15.81:0/22526070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c072430 msgr2=0x7fa98c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c072430 0x7fa98c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 -- 172.21.15.81:0/22526070 >> v1:172.21.15.106:6789/0 conn(0x7fa98c0c60a0 legacy=0x7fa98c15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.416+0000 7fa983400640 1 -- 172.21.15.81:0/22526070 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c0c5270 con 0x7fa98c15a040 2024-05-27T06:56:08.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa983400640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c0c5b60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa974009870 tx=0x7fa97401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a6813c1e45ba24 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:08.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa98b400640 1 -- 172.21.15.81:0/22526070 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740b0070 con 0x7fa98c15a040 2024-05-27T06:56:08.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa98b400640 1 -- 172.21.15.81:0/22526070 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9740ab600 con 0x7fa98c15a040 2024-05-27T06:56:08.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa98b400640 1 -- 172.21.15.81:0/22526070 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ab9d0 con 0x7fa98c15a040 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 -- 172.21.15.81:0/22526070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 msgr2=0x7fa98c0c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c0c5b60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa974009870 tx=0x7fa97401c560 comp rx=0 tx=0).stop 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 -- 172.21.15.81:0/22526070 shutdown_connections 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c0c5b60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 --2- 172.21.15.81:0/22526070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c072430 0x7fa98c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 -- 172.21.15.81:0/22526070 >> 172.21.15.81:0/22526070 conn(0x7fa98c070700 msgr2=0x7fa98c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 -- 172.21.15.81:0/22526070 shutdown_connections 2024-05-27T06:56:08.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.420+0000 7fa993000640 1 -- 172.21.15.81:0/22526070 wait complete. 2024-05-27T06:56:08.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 Processor -- start 2024-05-27T06:56:08.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 -- start start 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa98c072430 0x7fa98c14bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c0c60a0 0x7fa98c157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c0ce6f0 con 0x7fa98c15a040 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c0c7dd0 con 0x7fa98c072430 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa993000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c0c5610 con 0x7fa98c0c60a0 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa991000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa991000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44832/0 (socket says 172.21.15.81:44832) 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa991000640 1 -- 172.21.15.81:0/3965341312 learned_addr learned my addr 172.21.15.81:0/3965341312 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa98be00640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa98c072430 0x7fa98c14bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.424+0000 7fa983400640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c0c60a0 0x7fa98c157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 -- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c0c60a0 msgr2=0x7fa98c157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c0c60a0 0x7fa98c157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 -- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa98c072430 msgr2=0x7fa98c14bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa98c072430 0x7fa98c14bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 -- 172.21.15.81:0/3965341312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c1584d0 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa991000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c740 tx=0x7fa98400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984003b80 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98400ce50 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98400a850 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98c1544a0 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98c157350 con 0x7fa98c15a040 2024-05-27T06:56:08.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98c158220 con 0x7fa98c15a040 2024-05-27T06:56:08.419 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.428+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa98400a9f0 con 0x7fa98c15a040 2024-05-27T06:56:08.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.432+0000 7fa988c00640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 0x7fa968071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:08.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.432+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa98408f340 con 0x7fa98c15a040 2024-05-27T06:56:08.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.432+0000 7fa98be00640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 0x7fa968071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:08.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.432+0000 7fa98be00640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 0x7fa968071940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa98c070fe0 tx=0x7fa97c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:08.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.436+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa98405b8c0 con 0x7fa98c15a040 2024-05-27T06:56:08.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa98c06a560 con 0x7fa98c15a040 2024-05-27T06:56:08.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa988c00640 1 -- 172.21.15.81:0/3965341312 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7fa98405fbc0 con 0x7fa98c15a040 2024-05-27T06:56:08.617 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:08.617 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:08.617 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 msgr2=0x7fa968071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 0x7fa968071940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa98c070fe0 tx=0x7fa97c008040 comp rx=0 tx=0).stop 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 msgr2=0x7fa98c06ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c740 tx=0x7fa98400cc00 comp rx=0 tx=0).stop 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 shutdown_connections 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa98c0c60a0 0x7fa98c157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa96806f430 0x7fa968071940 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98c15a040 0x7fa98c06ce70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 --2- 172.21.15.81:0/3965341312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa98c072430 0x7fa98c14bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 >> 172.21.15.81:0/3965341312 conn(0x7fa98c070700 msgr2=0x7fa98c0ca190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 shutdown_connections 2024-05-27T06:56:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:08.628+0000 7fa993000640 1 -- 172.21.15.81:0/3965341312 wait complete. 2024-05-27T06:56:08.639 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:10.640 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:10.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 Processor -- start 2024-05-27T06:56:10.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 -- start start 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ac015fe70 0x7f6ac0160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac00c99f0 con 0x7f6ac0156bc0 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac0156fd0 con 0x7f6ac015fe70 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ac6000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac01607c0 con 0x7f6ac01650d0 2024-05-27T06:56:10.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ab6a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ac015fe70 0x7f6ac0160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6abf400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6ac0156bc0 0x7f6ac00cc2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45792/0 (socket says 172.21.15.81:45792) 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6abf400640 1 -- 172.21.15.81:0/3319795786 learned_addr learned my addr 172.21.15.81:0/3319795786 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6abfe00640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ab6a00640 1 -- 172.21.15.81:0/3319795786 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 msgr2=0x7f6ac01674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ab6a00640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ab6a00640 1 -- 172.21.15.81:0/3319795786 >> v1:172.21.15.81:6789/0 conn(0x7f6ac0156bc0 legacy=0x7f6ac00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6ab6a00640 1 -- 172.21.15.81:0/3319795786 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac015c980 con 0x7f6ac015fe70 2024-05-27T06:56:10.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.812+0000 7f6abea00640 1 -- 172.21.15.81:0/3319795786 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902370921 0 0) 0x7f6ac00c99f0 con 0x7f6ac0156bc0 2024-05-27T06:56:10.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ab6a00640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0160280 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80097f0 tx=0x7f6aa801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4954ce4e98172002 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:10.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6abea00640 1 -- 172.21.15.81:0/3319795786 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b5070 con 0x7f6ac015fe70 2024-05-27T06:56:10.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6abea00640 1 -- 172.21.15.81:0/3319795786 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa80a8990 con 0x7f6ac015fe70 2024-05-27T06:56:10.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6abea00640 1 -- 172.21.15.81:0/3319795786 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80a8d60 con 0x7f6ac015fe70 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 -- 172.21.15.81:0/3319795786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 msgr2=0x7f6ac0160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0160280 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80097f0 tx=0x7f6aa801ec90 comp rx=0 tx=0).stop 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 -- 172.21.15.81:0/3319795786 shutdown_connections 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 --2- 172.21.15.81:0/3319795786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0160280 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 -- 172.21.15.81:0/3319795786 >> 172.21.15.81:0/3319795786 conn(0x7f6ac0068cb0 msgr2=0x7f6ac00745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.816+0000 7f6ac6000640 1 -- 172.21.15.81:0/3319795786 shutdown_connections 2024-05-27T06:56:10.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 -- 172.21.15.81:0/3319795786 wait complete. 2024-05-27T06:56:10.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 Processor -- start 2024-05-27T06:56:10.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 -- start start 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 0x7f6ac014b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac015d220 con 0x7f6ac015fe70 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac015cf40 con 0x7f6ac0156bc0 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ac6000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac015d500 con 0x7f6ac01650d0 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44856/0 (socket says 172.21.15.81:44856) 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.820+0000 7f6ab6a00640 1 -- 172.21.15.81:0/505576959 learned_addr learned my addr 172.21.15.81:0/505576959 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 -- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 msgr2=0x7f6ac01486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:10.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 -- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 msgr2=0x7f6ac014b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6abf400640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 0x7f6ac014b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 0x7f6ac014b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 -- 172.21.15.81:0/505576959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac00c96d0 con 0x7f6ac015fe70 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6abf400640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 0x7f6ac014b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab6a00640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6aa801c8c0 tx=0x7f6aa801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80bf070 con 0x7f6ac015fe70 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa80b0680 con 0x7f6ac015fe70 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac014dba0 con 0x7f6ac015fe70 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b5040 con 0x7f6ac015fe70 2024-05-27T06:56:10.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac00c8ed0 con 0x7f6ac015fe70 2024-05-27T06:56:10.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac006a560 con 0x7f6ac015fe70 2024-05-27T06:56:10.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.824+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6aa80c2070 con 0x7f6ac015fe70 2024-05-27T06:56:10.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.828+0000 7f6ab7e00640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 0x7f6a9c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:10.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.828+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6aa812bb90 con 0x7f6ac015fe70 2024-05-27T06:56:10.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.828+0000 7f6abf400640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 0x7f6a9c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:10.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.828+0000 7f6abf400640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 0x7f6a9c071940 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007fd0 tx=0x7f6ab0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:10.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:10.832+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6aa80187f0 con 0x7f6ac015fe70 2024-05-27T06:56:11.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.020+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ac00caaf0 con 0x7f6ac015fe70 2024-05-27T06:56:11.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.020+0000 7f6ab7e00640 1 -- 172.21.15.81:0/505576959 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f6aa80f8800 con 0x7f6ac015fe70 2024-05-27T06:56:11.011 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:11.011 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:11.012 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 msgr2=0x7f6a9c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:11.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 0x7f6a9c071940 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0007fd0 tx=0x7f6ab0009040 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 msgr2=0x7f6ac0148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6aa801c8c0 tx=0x7f6aa801ba30 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 shutdown_connections 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ac01650d0 0x7f6ac01486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a9c06f430 0x7f6a9c071940 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac015fe70 0x7f6ac0148170 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 --2- 172.21.15.81:0/505576959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ac0156bc0 0x7f6ac014b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 >> 172.21.15.81:0/505576959 conn(0x7f6ac0068cb0 msgr2=0x7f6ac014ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 shutdown_connections 2024-05-27T06:56:11.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:11.024+0000 7f6ac6000640 1 -- 172.21.15.81:0/505576959 wait complete. 2024-05-27T06:56:11.033 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:11.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-05-27T06:56:13.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:13.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.203+0000 7f6dbe800640 1 Processor -- start 2024-05-27T06:56:13.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.203+0000 7f6dbe800640 1 -- start start 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6dbe800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6db81546f0 0x7f6db8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6dbe800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db815db00 con 0x7f6db80732f0 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6dbe800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6db8073860 con 0x7f6db815a510 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6dbe800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6db815e6f0 con 0x7f6db81546f0 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6db80732f0 0x7f6db80c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45802/0 (socket says 172.21.15.81:45802) 2024-05-27T06:56:13.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6a00640 1 -- 172.21.15.81:0/666311133 learned_addr learned my addr 172.21.15.81:0/666311133 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 --2- 172.21.15.81:0/666311133 >> v2:172.21.15.106:3300/0 conn(0x7f6db81546f0 0x7f6db8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442334940 0 0) 0x7f6db815db00 con 0x7f6db80732f0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d9c003530 con 0x7f6db80732f0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446234256 0 0) 0x7f6db8073860 con 0x7f6db815a510 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6db815db00 con 0x7f6db815a510 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2404667742 0 0) 0x7f6d9c003530 con 0x7f6db80732f0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 >> v2:172.21.15.106:3300/0 conn(0x7f6db81546f0 msgr2=0x7f6db8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 --2- 172.21.15.81:0/666311133 >> v2:172.21.15.106:3300/0 conn(0x7f6db81546f0 0x7f6db8072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 >> v1:172.21.15.106:6789/0 conn(0x7f6db815a510 legacy=0x7f6db815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db815ddc0 con 0x7f6db80732f0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f6da4001700 con 0x7f6db80732f0 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 >> v1:172.21.15.81:6789/0 conn(0x7f6db80732f0 legacy=0x7f6db80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6d9c004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d9c004810 0x7f6d9c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d9c005290 0x7f6d9c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6db815e6f0 con 0x7f6d9c003c40 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00c1a0 con 0x7f6d9c004810 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9c00c3a0 con 0x7f6d9c005290 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6d9c004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db6a00640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d9c004810 0x7f6d9c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7e00640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d9c005290 0x7f6d9c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 -- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d9c005290 msgr2=0x7f6d9c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6d9c005290 0x7f6d9c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 -- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d9c004810 msgr2=0x7f6d9c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d9c004810 0x7f6d9c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7400640 1 -- 172.21.15.81:0/666311133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d9c003530 con 0x7f6d9c003c40 2024-05-27T06:56:13.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.207+0000 7f6db7e00640 1 -- 172.21.15.81:0/666311133 reap_dead start 2024-05-27T06:56:13.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6db7400640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6d9c004090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009fc0 tx=0x7f6da801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba6c16af3316dc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:13.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80b2070 con 0x7f6d9c003c40 2024-05-27T06:56:13.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da80a5c90 con 0x7f6d9c003c40 2024-05-27T06:56:13.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6db6000640 1 -- 172.21.15.81:0/666311133 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8009c10 con 0x7f6d9c003c40 2024-05-27T06:56:13.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 -- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 msgr2=0x7f6d9c004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6d9c004090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009fc0 tx=0x7f6da801cc80 comp rx=0 tx=0).stop 2024-05-27T06:56:13.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 -- 172.21.15.81:0/666311133 shutdown_connections 2024-05-27T06:56:13.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 --2- 172.21.15.81:0/666311133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6d9c004090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 -- 172.21.15.81:0/666311133 >> 172.21.15.81:0/666311133 conn(0x7f6db8070700 msgr2=0x7f6db80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:13.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 -- 172.21.15.81:0/666311133 shutdown_connections 2024-05-27T06:56:13.203 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.211+0000 7f6dbe800640 1 -- 172.21.15.81:0/666311133 wait complete. 2024-05-27T06:56:13.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 Processor -- start 2024-05-27T06:56:13.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- start start 2024-05-27T06:56:13.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6db815c220 0x7f6db814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6db815c630 0x7f6db814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6db815e380 con 0x7f6d9c003c40 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6db80755b0 con 0x7f6db815c630 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6db8154b00 con 0x7f6db815c220 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44872/0 (socket says 172.21.15.81:44872) 2024-05-27T06:56:13.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 -- 172.21.15.81:0/1763925881 learned_addr learned my addr 172.21.15.81:0/1763925881 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db6a00640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6db815c220 0x7f6db814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7e00640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6db815c630 0x7f6db814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 -- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6db815c220 msgr2=0x7f6db814f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6db815c220 0x7f6db814f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 -- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6db815c630 msgr2=0x7f6db814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6db815c630 0x7f6db814f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 -- 172.21.15.81:0/1763925881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db80c5720 con 0x7f6d9c003c40 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6db7400640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6da80b0040 tx=0x7f6da8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80b2040 con 0x7f6d9c003c40 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da80a5a90 con 0x7f6d9c003c40 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db80c5a00 con 0x7f6d9c003c40 2024-05-27T06:56:13.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da80adc60 con 0x7f6d9c003c40 2024-05-27T06:56:13.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.215+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db81f6290 con 0x7f6d9c003c40 2024-05-27T06:56:13.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db81f64c0 con 0x7f6d9c003c40 2024-05-27T06:56:13.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6da80bf030 con 0x7f6d9c003c40 2024-05-27T06:56:13.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6daf400640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 0x7f6d98071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:13.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6da81299a0 con 0x7f6d9c003c40 2024-05-27T06:56:13.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6db6a00640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 0x7f6d98071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:13.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.219+0000 7f6db6a00640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 0x7f6d98071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6da4001c60 tx=0x7f6da4007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:13.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.223+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6db81f64c0 con 0x7f6d9c003c40 2024-05-27T06:56:13.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.411+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db806a560 con 0x7f6d9c003c40 2024-05-27T06:56:13.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.411+0000 7f6daf400640 1 -- 172.21.15.81:0/1763925881 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f6da80c6060 con 0x7f6d9c003c40 2024-05-27T06:56:13.404 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:13.404 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:13.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 msgr2=0x7f6d98071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 0x7f6d98071910 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6da4001c60 tx=0x7f6da4007040 comp rx=0 tx=0).stop 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 msgr2=0x7f6db80c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f6da80b0040 tx=0x7f6da8009750 comp rx=0 tx=0).stop 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 shutdown_connections 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6db815c220 0x7f6db814f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6d9806f400 0x7f6d98071910 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6db815c630 0x7f6db814f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 --2- 172.21.15.81:0/1763925881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d9c003c40 0x7f6db80c50a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:13.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 >> 172.21.15.81:0/1763925881 conn(0x7f6db8070700 msgr2=0x7f6db815b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:13.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 shutdown_connections 2024-05-27T06:56:13.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:13.415+0000 7f6dbe800640 1 -- 172.21.15.81:0/1763925881 wait complete. 2024-05-27T06:56:13.425 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:15.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:15.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 Processor -- start 2024-05-27T06:56:15.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 -- start start 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd51815d2b0 0x7fd5180c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5180c8ca0 con 0x7fd5181650c0 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd518156f90 con 0x7fd5180cabf0 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd51d800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd5180c93f0 con 0x7fd51815d2b0 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd517400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd5181650c0 0x7fd5181674e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35044/0 (socket says 172.21.15.81:35044) 2024-05-27T06:56:15.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd517400640 1 -- 172.21.15.81:0/428663113 learned_addr learned my addr 172.21.15.81:0/428663113 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.595+0000 7fd516a00640 1 --2- 172.21.15.81:0/428663113 >> v2:172.21.15.106:3300/0 conn(0x7fd51815d2b0 0x7fd5180c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909606310 0 0) 0x7fd5180c8ca0 con 0x7fd5181650c0 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4fc003530 con 0x7fd5181650c0 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2322429739 0 0) 0x7fd4fc003530 con 0x7fd5181650c0 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 >> v2:172.21.15.106:3300/0 conn(0x7fd51815d2b0 msgr2=0x7fd5180c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 --2- 172.21.15.81:0/428663113 >> v2:172.21.15.106:3300/0 conn(0x7fd51815d2b0 0x7fd5180c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 >> v1:172.21.15.106:6789/0 conn(0x7fd5180cabf0 legacy=0x7fd51806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5180c9640 con 0x7fd5181650c0 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fd510001700 con 0x7fd5181650c0 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 >> v1:172.21.15.81:6789/0 conn(0x7fd5181650c0 legacy=0x7fd5181674e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4fc003a60 0x7fd4fc003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4fc004810 0x7fd4fc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd4fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180c93f0 con 0x7fd4fc005260 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc00c1f0 con 0x7fd4fc004810 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd4fc00c3f0 con 0x7fd4fc003a60 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd4fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd516a00640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4fc003a60 0x7fd4fc003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd50e000640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4fc004810 0x7fd4fc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 -- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4fc003a60 msgr2=0x7fd4fc003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd4fc003a60 0x7fd4fc003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 -- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4fc004810 msgr2=0x7fd4fc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4fc004810 0x7fd4fc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.599+0000 7fd517400640 1 -- 172.21.15.81:0/428663113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc003530 con 0x7fd4fc005260 2024-05-27T06:56:15.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd517400640 1 -- 172.21.15.81:0/428663113 reap_dead start 2024-05-27T06:56:15.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd517400640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd4fc00b9d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c210 tx=0x7fd51001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cef7954485063cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:15.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510007cf0 con 0x7fd4fc005260 2024-05-27T06:56:15.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5100a5b40 con 0x7fd4fc005260 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd516000640 1 -- 172.21.15.81:0/428663113 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5100058e0 con 0x7fd4fc005260 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 msgr2=0x7fd4fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd4fc00b9d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd51000c210 tx=0x7fd51001cba0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- 172.21.15.81:0/428663113 shutdown_connections 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 --2- 172.21.15.81:0/428663113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd4fc00b9d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- 172.21.15.81:0/428663113 >> 172.21.15.81:0/428663113 conn(0x7fd5180ce8d0 msgr2=0x7fd5180cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- 172.21.15.81:0/428663113 shutdown_connections 2024-05-27T06:56:15.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- 172.21.15.81:0/428663113 wait complete. 2024-05-27T06:56:15.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 Processor -- start 2024-05-27T06:56:15.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.603+0000 7fd51d800640 1 -- start start 2024-05-27T06:56:15.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd518156250 0x7fd518159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 0x7fd51814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180ca3f0 con 0x7fd4fc005260 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180ca1a0 con 0x7fd518159180 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd5180ca6d0 con 0x7fd518156250 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48298/0 (socket says 172.21.15.81:48298) 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 -- 172.21.15.81:0/2503618369 learned_addr learned my addr 172.21.15.81:0/2503618369 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd50e000640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd518156250 0x7fd518159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd517400640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 0x7fd51814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 -- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd518156250 msgr2=0x7fd518159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd518156250 0x7fd518159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 -- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 msgr2=0x7fd51814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 0x7fd51814a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 -- 172.21.15.81:0/2503618369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5181f7080 con 0x7fd4fc005260 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd517400640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 0x7fd51814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd516a00640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd508009f90 tx=0x7fd50800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508015430 con 0x7fd4fc005260 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5080155d0 con 0x7fd4fc005260 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5080159c0 con 0x7fd4fc005260 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5181573f0 con 0x7fd4fc005260 2024-05-27T06:56:15.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5181f6620 con 0x7fd4fc005260 2024-05-27T06:56:15.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.607+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd51806a560 con 0x7fd4fc005260 2024-05-27T06:56:15.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.611+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd508015b60 con 0x7fd4fc005260 2024-05-27T06:56:15.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.611+0000 7fd50f400640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 0x7fd4f4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:15.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.611+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd5080906a0 con 0x7fd4fc005260 2024-05-27T06:56:15.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.611+0000 7fd50e000640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 0x7fd4f4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:15.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.611+0000 7fd50e000640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 0x7fd4f4071940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd518153ac0 tx=0x7fd500009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:15.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.615+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd508019030 con 0x7fd4fc005260 2024-05-27T06:56:15.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd518154090 con 0x7fd4fc005260 2024-05-27T06:56:15.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd50f400640 1 -- 172.21.15.81:0/2503618369 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7fd508010040 con 0x7fd4fc005260 2024-05-27T06:56:15.798 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:15.798 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:15.799 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 msgr2=0x7fd4f4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 0x7fd4f4071940 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd518153ac0 tx=0x7fd500009040 comp rx=0 tx=0).stop 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 msgr2=0x7fd5181f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.807+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd508009f90 tx=0x7fd50800f580 comp rx=0 tx=0).stop 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 shutdown_connections 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd518156250 0x7fd518159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd4f406f430 0x7fd4f4071940 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd518159180 0x7fd51814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 --2- 172.21.15.81:0/2503618369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4fc005260 0x7fd5181f6b40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:15.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 >> 172.21.15.81:0/2503618369 conn(0x7fd5180ce8d0 msgr2=0x7fd51814d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:15.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 shutdown_connections 2024-05-27T06:56:15.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:15.811+0000 7fd51d800640 1 -- 172.21.15.81:0/2503618369 wait complete. 2024-05-27T06:56:15.820 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:17.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-05-27T06:56:17.820 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:17.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 Processor -- start 2024-05-27T06:56:17.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 -- start start 2024-05-27T06:56:17.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6afc154790 0x7f6afc0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6afc0ca380 con 0x7f6afc071f20 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6afc0ca660 con 0x7f6afc0caef0 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b02800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6afc0ca940 con 0x7f6afc154790 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b01e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6afc071f20 0x7f6afc164fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35048/0 (socket says 172.21.15.81:35048) 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b01e00640 1 -- 172.21.15.81:0/3008345900 learned_addr learned my addr 172.21.15.81:0/3008345900 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:17.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b00a00640 1 --2- 172.21.15.81:0/3008345900 >> v2:172.21.15.81:3300/0 conn(0x7f6afc154790 0x7f6afc0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:17.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b00a00640 1 -- 172.21.15.81:0/3008345900 >> v1:172.21.15.106:6789/0 conn(0x7f6afc0caef0 legacy=0x7f6afc15e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:17.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b00a00640 1 -- 172.21.15.81:0/3008345900 >> v1:172.21.15.81:6789/0 conn(0x7f6afc071f20 legacy=0x7f6afc164fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:17.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:17.999+0000 7f6b00a00640 1 -- 172.21.15.81:0/3008345900 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6afc15de20 con 0x7f6afc154790 2024-05-27T06:56:17.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b00a00640 1 --2- 172.21.15.81:0/3008345900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc154790 0x7f6afc0c9710 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80097c0 tx=0x7f6ae801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1228000337921fca server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:17.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6afbe00640 1 -- 172.21.15.81:0/3008345900 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae80b5070 con 0x7f6afc154790 2024-05-27T06:56:17.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6afbe00640 1 -- 172.21.15.81:0/3008345900 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae80a8990 con 0x7f6afc154790 2024-05-27T06:56:17.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6afbe00640 1 -- 172.21.15.81:0/3008345900 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae80a8d60 con 0x7f6afc154790 2024-05-27T06:56:17.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 -- 172.21.15.81:0/3008345900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc154790 msgr2=0x7f6afc0c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:17.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 --2- 172.21.15.81:0/3008345900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc154790 0x7f6afc0c9710 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80097c0 tx=0x7f6ae801ec90 comp rx=0 tx=0).stop 2024-05-27T06:56:17.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 -- 172.21.15.81:0/3008345900 shutdown_connections 2024-05-27T06:56:17.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 --2- 172.21.15.81:0/3008345900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc154790 0x7f6afc0c9710 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:17.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 -- 172.21.15.81:0/3008345900 >> 172.21.15.81:0/3008345900 conn(0x7f6afc068d90 msgr2=0x7f6afc0cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:17.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 -- 172.21.15.81:0/3008345900 shutdown_connections 2024-05-27T06:56:17.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.003+0000 7f6b02800640 1 -- 172.21.15.81:0/3008345900 wait complete. 2024-05-27T06:56:17.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 Processor -- start 2024-05-27T06:56:17.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 -- start start 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6afc0caef0 0x7f6afc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6afc154790 0x7f6afc145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc154ba0 con 0x7f6afc071f20 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc0cbfe0 con 0x7f6afc154790 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6afc154d20 con 0x7f6afc0caef0 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48314/0 (socket says 172.21.15.81:48314) 2024-05-27T06:56:17.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 -- 172.21.15.81:0/3671182089 learned_addr learned my addr 172.21.15.81:0/3671182089 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01e00640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6afc154790 0x7f6afc145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b00a00640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6afc0caef0 0x7f6afc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 -- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6afc0caef0 msgr2=0x7f6afc1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6afc0caef0 0x7f6afc1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 -- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6afc154790 msgr2=0x7f6afc145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6afc154790 0x7f6afc145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 -- 172.21.15.81:0/3671182089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6afc1f5260 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b01400640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6af000c3f0 tx=0x7f6af000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af0017070 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af00052d0 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af0003820 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.007+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6afc1f0dd0 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6afc1f1090 con 0x7f6afc071f20 2024-05-27T06:56:18.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6afc050870 con 0x7f6afc071f20 2024-05-27T06:56:18.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6af001a070 con 0x7f6afc071f20 2024-05-27T06:56:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6af9600640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 0x7f6adc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6af008e800 con 0x7f6afc071f20 2024-05-27T06:56:18.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.011+0000 7f6b00a00640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 0x7f6adc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:18.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.015+0000 7f6b00a00640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 0x7f6adc071910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80b2000 tx=0x7f6ae8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:18.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.019+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6af0005480 con 0x7f6afc071f20 2024-05-27T06:56:18.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.207+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6afc158040 con 0x7f6afc071f20 2024-05-27T06:56:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.207+0000 7f6af9600640 1 -- 172.21.15.81:0/3671182089 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6087 (secure 0 0 0) 0x7f6af005ae10 con 0x7f6afc071f20 2024-05-27T06:56:18.198 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:18.198 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:55:53.380697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[4870,4949,5049,5184]},"id":1}]} 2024-05-27T06:56:18.198 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 48 2024-05-27T06:56:18.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 msgr2=0x7f6adc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:18.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 0x7f6adc071910 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80b2000 tx=0x7f6ae8008040 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 msgr2=0x7f6afc1f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6af000c3f0 tx=0x7f6af000c8b0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 shutdown_connections 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6afc0caef0 0x7f6afc1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6adc06f400 0x7f6adc071910 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ae80b2000 tx=0x7f6ae8008040 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6afc154790 0x7f6afc145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 --2- 172.21.15.81:0/3671182089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc071f20 0x7f6afc1f4d20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 >> 172.21.15.81:0/3671182089 conn(0x7f6afc068d90 msgr2=0x7f6afc0c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 shutdown_connections 2024-05-27T06:56:18.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:18.211+0000 7f6b02800640 1 -- 172.21.15.81:0/3671182089 wait complete. 2024-05-27T06:56:18.221 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:55:53.380697+0000", "qdb_cluster": [ 4870, 4949, 5049, 5184 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:56:20.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:20.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 Processor -- start 2024-05-27T06:56:20.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 -- start start 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb7b0160470 0x7fb7b0160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b00cd390 con 0x7fb7b0156900 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b006f120 con 0x7fb7b006c790 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7b5400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b015fd00 con 0x7fb7b0160470 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7a5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb7b0160470 0x7fb7b0160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:20.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7ae800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb7b0156900 0x7fb7b00cc660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35054/0 (socket says 172.21.15.81:35054) 2024-05-27T06:56:20.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7ae800640 1 -- 172.21.15.81:0/3322177422 learned_addr learned my addr 172.21.15.81:0/3322177422 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:20.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7a5e00640 1 -- 172.21.15.81:0/3322177422 >> v1:172.21.15.106:6789/0 conn(0x7fb7b006c790 legacy=0x7fb7b0167090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7a5e00640 1 -- 172.21.15.81:0/3322177422 >> v1:172.21.15.81:6789/0 conn(0x7fb7b0156900 legacy=0x7fb7b00cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7a5e00640 1 -- 172.21.15.81:0/3322177422 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b006f400 con 0x7fb7b0160470 2024-05-27T06:56:20.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.391+0000 7fb7ade00640 1 -- 172.21.15.81:0/3322177422 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460686918 0 0) 0x7fb7b00cd390 con 0x7fb7b0156900 2024-05-27T06:56:20.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7a5e00640 1 --2- 172.21.15.81:0/3322177422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b0160470 0x7fb7b0160880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb798009de0 tx=0x7fb79801f190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a187827508f047b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:20.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7ade00640 1 -- 172.21.15.81:0/3322177422 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7980b4070 con 0x7fb7b0160470 2024-05-27T06:56:20.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7ade00640 1 -- 172.21.15.81:0/3322177422 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7980af600 con 0x7fb7b0160470 2024-05-27T06:56:20.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7ade00640 1 -- 172.21.15.81:0/3322177422 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7980af9d0 con 0x7fb7b0160470 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 -- 172.21.15.81:0/3322177422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b0160470 msgr2=0x7fb7b0160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 --2- 172.21.15.81:0/3322177422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b0160470 0x7fb7b0160880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb798009de0 tx=0x7fb79801f190 comp rx=0 tx=0).stop 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 -- 172.21.15.81:0/3322177422 shutdown_connections 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 --2- 172.21.15.81:0/3322177422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b0160470 0x7fb7b0160880 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 -- 172.21.15.81:0/3322177422 >> 172.21.15.81:0/3322177422 conn(0x7fb7b00cb2e0 msgr2=0x7fb7b00cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 -- 172.21.15.81:0/3322177422 shutdown_connections 2024-05-27T06:56:20.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.395+0000 7fb7b5400640 1 -- 172.21.15.81:0/3322177422 wait complete. 2024-05-27T06:56:20.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 Processor -- start 2024-05-27T06:56:20.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 -- start start 2024-05-27T06:56:20.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:20.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7b0156900 0x7fb7b01f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0160470 0x7fb7b00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b006f9c0 con 0x7fb7b006c790 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b015ffe0 con 0x7fb7b0160470 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7b5400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b006d5b0 con 0x7fb7b0156900 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7ae800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7ae800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48336/0 (socket says 172.21.15.81:48336) 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7ae800640 1 -- 172.21.15.81:0/247860138 learned_addr learned my addr 172.21.15.81:0/247860138 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.399+0000 7fb7a5e00640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7b0156900 0x7fb7b01f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7af200640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0160470 0x7fb7b00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 -- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7b0156900 msgr2=0x7fb7b01f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7b0156900 0x7fb7b01f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 -- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0160470 msgr2=0x7fb7b00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0160470 0x7fb7b00c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 -- 172.21.15.81:0/247860138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b01f3280 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7ae800640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000c3c0 tx=0x7fb7a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a00074e0 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a00047f0 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b015a720 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0007680 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b015ace0 con 0x7fb7b006c790 2024-05-27T06:56:20.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b006a560 con 0x7fb7b006c790 2024-05-27T06:56:20.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb7a0011070 con 0x7fb7b006c790 2024-05-27T06:56:20.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.403+0000 7fb7a7400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 0x7fb78c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:20.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.407+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb7a008f090 con 0x7fb7b006c790 2024-05-27T06:56:20.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.407+0000 7fb7a5e00640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 0x7fb78c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:20.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.407+0000 7fb7a5e00640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 0x7fb78c071910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb7980b1000 tx=0x7fb7980aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.411+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb7a005bcf0 con 0x7fb7b006c790 2024-05-27T06:56:20.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.599+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7b0152190 con 0x7fb7b006c790 2024-05-27T06:56:20.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.599+0000 7fb7a7400640 1 -- 172.21.15.81:0/247860138 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fb7a005b5f0 con 0x7fb7b006c790 2024-05-27T06:56:20.592 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:20.592 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:20.592 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 msgr2=0x7fb78c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 0x7fb78c071910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb7980b1000 tx=0x7fb7980aa000 comp rx=0 tx=0).stop 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 msgr2=0x7fb7b01e80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000c3c0 tx=0x7fb7a000c880 comp rx=0 tx=0).stop 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 shutdown_connections 2024-05-27T06:56:20.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb7b0156900 0x7fb7b01f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb78c06f400 0x7fb78c071910 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0160470 0x7fb7b00c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 --2- 172.21.15.81:0/247860138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7b006c790 0x7fb7b01e80e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 >> 172.21.15.81:0/247860138 conn(0x7fb7b00cb2e0 msgr2=0x7fb7b01514f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 shutdown_connections 2024-05-27T06:56:20.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:20.603+0000 7fb7b5400640 1 -- 172.21.15.81:0/247860138 wait complete. 2024-05-27T06:56:20.614 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:22.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:22.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.787+0000 7f665ac00640 1 Processor -- start 2024-05-27T06:56:22.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.787+0000 7f665ac00640 1 -- start start 2024-05-27T06:56:22.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f665ac00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f665415fd20 0x7f6654156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f665ac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f665ac00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6654160130 con 0x7f66540ca900 2024-05-27T06:56:22.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f665ac00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f66540c9070 con 0x7f665415fd20 2024-05-27T06:56:22.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f665ac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6654074890 con 0x7f66540c89a0 2024-05-27T06:56:22.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f664b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6653e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f665415fd20 0x7f6654156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6658c00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f66540ca900 0x7f66540cad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:54742/0 (socket says 172.21.15.81:54742) 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6658c00640 1 -- 172.21.15.81:0/4229705539 learned_addr learned my addr 172.21.15.81:0/4229705539 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f664b400640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:51516/0 (socket says 172.21.15.81:51516) 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6653e00640 1 -- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 msgr2=0x7f6654072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:22.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6653e00640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6653e00640 1 -- 172.21.15.81:0/4229705539 >> v1:172.21.15.106:6789/0 conn(0x7f66540ca900 legacy=0x7f66540cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f6653e00640 1 -- 172.21.15.81:0/4229705539 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66540c94c0 con 0x7f665415fd20 2024-05-27T06:56:22.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.791+0000 7f664b400640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:56:22.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f6653e00640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f6654156a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6644009d90 tx=0x7f664401c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9b1780abdd9760 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:22.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f6653400640 1 -- 172.21.15.81:0/4229705539 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66440b2070 con 0x7f665415fd20 2024-05-27T06:56:22.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f6653400640 1 -- 172.21.15.81:0/4229705539 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66440a5820 con 0x7f665415fd20 2024-05-27T06:56:22.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f6653400640 1 -- 172.21.15.81:0/4229705539 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66440a5bf0 con 0x7f665415fd20 2024-05-27T06:56:22.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 -- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 msgr2=0x7f6654156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f6654156a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6644009d90 tx=0x7f664401c800 comp rx=0 tx=0).stop 2024-05-27T06:56:22.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 -- 172.21.15.81:0/4229705539 shutdown_connections 2024-05-27T06:56:22.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540c89a0 0x7f6654072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 --2- 172.21.15.81:0/4229705539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f6654156a30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 -- 172.21.15.81:0/4229705539 >> 172.21.15.81:0/4229705539 conn(0x7f665406f230 msgr2=0x7f6654070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:22.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 -- 172.21.15.81:0/4229705539 shutdown_connections 2024-05-27T06:56:22.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.795+0000 7f665ac00640 1 -- 172.21.15.81:0/4229705539 wait complete. 2024-05-27T06:56:22.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 Processor -- start 2024-05-27T06:56:22.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- start start 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540ca900 0x7f66541f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f66541f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6654073fe0 con 0x7f66540c89a0 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6654073d00 con 0x7f665415fd20 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f66540c9760 con 0x7f66540ca900 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48352/0 (socket says 172.21.15.81:48352) 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 -- 172.21.15.81:0/4185255009 learned_addr learned my addr 172.21.15.81:0/4185255009 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:22.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6658c00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f66541f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f664b400640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540ca900 0x7f66541f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 -- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540ca900 msgr2=0x7f66541f6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540ca900 0x7f66541f6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 -- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 msgr2=0x7f66541f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f66541f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 -- 172.21.15.81:0/4185255009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665415f0a0 con 0x7f66540c89a0 2024-05-27T06:56:22.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6658c00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f66541f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6653e00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f66440b0040 tx=0x7f66440195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66440b2070 con 0x7f66540c89a0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66440bc070 con 0x7f66540c89a0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6644009450 con 0x7f66540c89a0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66541f7a40 con 0x7f66540c89a0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.799+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f665415f7a0 con 0x7f66540c89a0 2024-05-27T06:56:22.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665415f270 con 0x7f66540c89a0 2024-05-27T06:56:22.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f66440adc20 con 0x7f66540c89a0 2024-05-27T06:56:22.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f6650c00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 0x7f6630071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:22.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6644129fe0 con 0x7f66540c89a0 2024-05-27T06:56:22.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f664b400640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 0x7f6630071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:22.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.803+0000 7f664b400640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 0x7f6630071910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f66540771b0 tx=0x7f663c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:22.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.807+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6644016480 con 0x7f66540c89a0 2024-05-27T06:56:22.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.995+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f665406a560 con 0x7f66540c89a0 2024-05-27T06:56:22.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f6650c00640 1 -- 172.21.15.81:0/4185255009 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f66440f6d70 con 0x7f66540c89a0 2024-05-27T06:56:22.989 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:22.989 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:22.989 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 msgr2=0x7f6630071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 0x7f6630071910 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f66540771b0 tx=0x7f663c008040 comp rx=0 tx=0).stop 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 msgr2=0x7f665415eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f66440b0040 tx=0x7f66440195a0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 shutdown_connections 2024-05-27T06:56:22.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f66540ca900 0x7f66541f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f663006f400 0x7f6630071910 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f665415fd20 0x7f66541f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 --2- 172.21.15.81:0/4185255009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66540c89a0 0x7f665415eb60 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 >> 172.21.15.81:0/4185255009 conn(0x7f665406f230 msgr2=0x7f66540c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 shutdown_connections 2024-05-27T06:56:22.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:22.999+0000 7f665ac00640 1 -- 172.21.15.81:0/4185255009 wait complete. 2024-05-27T06:56:23.011 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:24.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~20s 2024-05-27T06:56:25.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:25.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 Processor -- start 2024-05-27T06:56:25.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 -- start start 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3748156a50 0x7f3748159e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3748070c20 0x7f3748071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37480cabd0 con 0x7f37480c9dd0 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f374815a380 con 0x7f3748156a50 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f374ce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3748156e60 con 0x7f3748070c20 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f37480c9dd0 0x7f37480ca1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35070/0 (socket says 172.21.15.81:35070) 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3748156a50 0x7f3748159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747e00640 1 -- 172.21.15.81:0/4076490371 learned_addr learned my addr 172.21.15.81:0/4076490371 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3746a00640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3748070c20 0x7f3748071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:25.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747400640 1 -- 172.21.15.81:0/4076490371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3748070c20 msgr2=0x7f3748071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:25.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747400640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3748070c20 0x7f3748071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747400640 1 -- 172.21.15.81:0/4076490371 >> v1:172.21.15.81:6789/0 conn(0x7f37480c9dd0 legacy=0x7f37480ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3746000640 1 -- 172.21.15.81:0/4076490371 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2285467712 0 0) 0x7f37480cabd0 con 0x7f37480c9dd0 2024-05-27T06:56:25.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.183+0000 7f3747400640 1 -- 172.21.15.81:0/4076490371 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37480cae70 con 0x7f3748156a50 2024-05-27T06:56:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f3747400640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f3748159e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f37380097c0 tx=0x7f373801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a3a1459cd4df17 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f3746000640 1 -- 172.21.15.81:0/4076490371 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380b0070 con 0x7f3748156a50 2024-05-27T06:56:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f3746000640 1 -- 172.21.15.81:0/4076490371 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3738004230 con 0x7f3748156a50 2024-05-27T06:56:25.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f3746000640 1 -- 172.21.15.81:0/4076490371 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738003620 con 0x7f3748156a50 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 -- 172.21.15.81:0/4076490371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 msgr2=0x7f3748159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f3748159e40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f37380097c0 tx=0x7f373801c440 comp rx=0 tx=0).stop 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 -- 172.21.15.81:0/4076490371 shutdown_connections 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3748070c20 0x7f3748071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 --2- 172.21.15.81:0/4076490371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f3748159e40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 -- 172.21.15.81:0/4076490371 >> 172.21.15.81:0/4076490371 conn(0x7f374806dea0 msgr2=0x7f37480ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 -- 172.21.15.81:0/4076490371 shutdown_connections 2024-05-27T06:56:25.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.187+0000 7f374ce00640 1 -- 172.21.15.81:0/4076490371 wait complete. 2024-05-27T06:56:25.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 Processor -- start 2024-05-27T06:56:25.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 -- start start 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 0x7f37481437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37480c9dd0 0x7f37481f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f37480cf6f0 con 0x7f3748156a50 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f37480cf570 con 0x7f3748070c20 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f374ce00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3748166480 con 0x7f37480c9dd0 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48358/0 (socket says 172.21.15.81:48358) 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 -- 172.21.15.81:0/4224758360 learned_addr learned my addr 172.21.15.81:0/4224758360 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:25.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747400640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 0x7f37481437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 -- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37480c9dd0 msgr2=0x7f37481f8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37480c9dd0 0x7f37481f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 -- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 msgr2=0x7f37481437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 0x7f37481437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 -- 172.21.15.81:0/4224758360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3748143cf0 con 0x7f3748156a50 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747400640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 0x7f37481437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f3747e00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f374000c680 tx=0x7f374000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.191+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740018070 con 0x7f3748156a50 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3740004d70 con 0x7f3748156a50 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37481f98a0 con 0x7f3748156a50 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374000a880 con 0x7f3748156a50 2024-05-27T06:56:25.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374814a3f0 con 0x7f3748156a50 2024-05-27T06:56:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3748050870 con 0x7f3748156a50 2024-05-27T06:56:25.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f374001b070 con 0x7f3748156a50 2024-05-27T06:56:25.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f373f400640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 0x7f3728071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:25.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f374008eab0 con 0x7f3748156a50 2024-05-27T06:56:25.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.195+0000 7f3747400640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 0x7f3728071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:25.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.199+0000 7f3747400640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 0x7f3728071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f37380ad000 tx=0x7f37380a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:25.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.203+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f374005b710 con 0x7f3748156a50 2024-05-27T06:56:25.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.391+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f374806d3c0 con 0x7f3748156a50 2024-05-27T06:56:25.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.391+0000 7f373f400640 1 -- 172.21.15.81:0/4224758360 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f374005b010 con 0x7f3748156a50 2024-05-27T06:56:25.383 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:25.383 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:25.383 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:25.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 msgr2=0x7f3728071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 0x7f3728071910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f37380ad000 tx=0x7f37380a6000 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 msgr2=0x7f374813e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f374000c680 tx=0x7f374000cb40 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 shutdown_connections 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f37480c9dd0 0x7f37481f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f372806f400 0x7f3728071910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3748156a50 0x7f374813e140 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 --2- 172.21.15.81:0/4224758360 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3748070c20 0x7f37481437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 >> 172.21.15.81:0/4224758360 conn(0x7f374806dea0 msgr2=0x7f3748071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 shutdown_connections 2024-05-27T06:56:25.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:25.395+0000 7f374ce00640 1 -- 172.21.15.81:0/4224758360 wait complete. 2024-05-27T06:56:25.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:27.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:27.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 Processor -- start 2024-05-27T06:56:27.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 -- start start 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c00735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f39c00cbfc0 0x7f39c00cf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c00caab0 con 0x7f39c00cf060 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f39c00c99c0 con 0x7f39c00cbfc0 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c7a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f39c00cad20 con 0x7f39c0159f40 2024-05-27T06:56:27.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39b7e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f39c00cf060 0x7f39c00cb8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50836/0 (socket says 172.21.15.81:50836) 2024-05-27T06:56:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39b7e00640 1 -- 172.21.15.81:0/3239715064 learned_addr learned my addr 172.21.15.81:0/3239715064 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c5a00640 1 --2- 172.21.15.81:0/3239715064 >> v2:172.21.15.81:3300/0 conn(0x7f39c00cbfc0 0x7f39c00cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c5000640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c00735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39bfe00640 1 -- 172.21.15.81:0/3239715064 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338296066 0 0) 0x7f39c00caab0 con 0x7f39c00cf060 2024-05-27T06:56:27.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c5a00640 1 -- 172.21.15.81:0/3239715064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 msgr2=0x7f39c00735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.575+0000 7f39c5a00640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c00735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39c5a00640 1 -- 172.21.15.81:0/3239715064 >> v1:172.21.15.81:6789/0 conn(0x7f39c00cf060 legacy=0x7f39c00cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39c5a00640 1 -- 172.21.15.81:0/3239715064 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c00cb000 con 0x7f39c00cbfc0 2024-05-27T06:56:27.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39c5a00640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cbfc0 0x7f39c00cf470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009d90 tx=0x7f39b801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6becff57795861b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:27.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39bfe00640 1 -- 172.21.15.81:0/3239715064 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80b4070 con 0x7f39c00cbfc0 2024-05-27T06:56:27.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39bfe00640 1 -- 172.21.15.81:0/3239715064 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b80af600 con 0x7f39c00cbfc0 2024-05-27T06:56:27.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39bfe00640 1 -- 172.21.15.81:0/3239715064 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80af9d0 con 0x7f39c00cbfc0 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39c7a00640 1 -- 172.21.15.81:0/3239715064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cbfc0 msgr2=0x7f39c00cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.579+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cbfc0 0x7f39c00cf470 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009d90 tx=0x7f39b801f150 comp rx=0 tx=0).stop 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- 172.21.15.81:0/3239715064 shutdown_connections 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cbfc0 0x7f39c00cf470 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3239715064 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c00735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- 172.21.15.81:0/3239715064 >> 172.21.15.81:0/3239715064 conn(0x7f39c006dca0 msgr2=0x7f39c006e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- 172.21.15.81:0/3239715064 shutdown_connections 2024-05-27T06:56:27.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- 172.21.15.81:0/3239715064 wait complete. 2024-05-27T06:56:27.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 Processor -- start 2024-05-27T06:56:27.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- start start 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 0x7f39c0151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c0152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0073ae0 con 0x7f39c00cf060 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f39c00c9ca0 con 0x7f39c00cbfc0 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.583+0000 7f39c7a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f39c0073c60 con 0x7f39c0159f40 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49686/0 (socket says 172.21.15.81:49686) 2024-05-27T06:56:27.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 -- 172.21.15.81:0/3145173290 learned_addr learned my addr 172.21.15.81:0/3145173290 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c5a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c0152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c5000640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 0x7f39c0151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 -- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 msgr2=0x7f39c0152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c0152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 -- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 msgr2=0x7f39c0151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 0x7f39c0151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 -- 172.21.15.81:0/3145173290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c0150ee0 con 0x7f39c00cf060 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c5000640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 0x7f39c0151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39b7e00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f39a800c460 tx=0x7f39a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8018070 con 0x7f39c00cf060 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a8007500 con 0x7f39c00cf060 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c01e30b0 con 0x7f39c00cf060 2024-05-27T06:56:27.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8004a30 con 0x7f39c00cf060 2024-05-27T06:56:27.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c01e3670 con 0x7f39c00cf060 2024-05-27T06:56:27.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c006a560 con 0x7f39c00cf060 2024-05-27T06:56:27.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.587+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f39a801b070 con 0x7f39c00cf060 2024-05-27T06:56:27.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.591+0000 7f39bd600640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 0x7f399c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:27.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.591+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f39a808e730 con 0x7f39c00cf060 2024-05-27T06:56:27.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.591+0000 7f39c5000640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 0x7f399c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:27.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.591+0000 7f39c5000640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 0x7f399c071910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39b000b9f0 tx=0x7f39b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:27.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.595+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f39a805ad40 con 0x7f39c00cf060 2024-05-27T06:56:27.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.787+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39c00ca5e0 con 0x7f39c00cf060 2024-05-27T06:56:27.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.787+0000 7f39bd600640 1 -- 172.21.15.81:0/3145173290 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f39a805f040 con 0x7f39c00cf060 2024-05-27T06:56:27.779 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:27.779 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:27.779 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 msgr2=0x7f399c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 0x7f399c071910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39b000b9f0 tx=0x7f39b0008040 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 msgr2=0x7f39c01509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f39a800c460 tx=0x7f39a800c920 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 shutdown_connections 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39c0159f40 0x7f39c0152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f399c06f400 0x7f399c071910 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39c00cf060 0x7f39c01509a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 --2- 172.21.15.81:0/3145173290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39c00cbfc0 0x7f39c0151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:27.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 >> 172.21.15.81:0/3145173290 conn(0x7f39c006dca0 msgr2=0x7f39c014fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:27.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 shutdown_connections 2024-05-27T06:56:27.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:27.791+0000 7f39c7a00640 1 -- 172.21.15.81:0/3145173290 wait complete. 2024-05-27T06:56:27.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:29.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:29.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.971+0000 7f5ae3600640 1 Processor -- start 2024-05-27T06:56:29.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.971+0000 7f5ae3600640 1 -- start start 2024-05-27T06:56:29.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae3600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae3600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 0x7f5adc0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae3600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5adc0c9f80 con 0x7f5adc15cb80 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae3600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5adc0c58b0 con 0x7f5adc0c8880 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae3600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc0c83f0 con 0x7f5adc06c790 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae0c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5adc15cb80 0x7f5adc152cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50850/0 (socket says 172.21.15.81:50850) 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae0c00640 1 -- 172.21.15.81:0/3118143551 learned_addr learned my addr 172.21.15.81:0/3118143551 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae1600640 1 --2- 172.21.15.81:0/3118143551 >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 0x7f5adc0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/3118143551 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075650350 0 0) 0x7f5adc0c9f80 con 0x7f5adc15cb80 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ac0005740 con 0x7f5adc15cb80 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1785965396 0 0) 0x7f5ac0005740 con 0x7f5adc15cb80 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ad3e00640 1 -- 172.21.15.81:0/3118143551 >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 msgr2=0x7f5adc0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/3118143551 >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 0x7f5adc0c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ad3e00640 1 -- 172.21.15.81:0/3118143551 >> v1:172.21.15.81:6789/0 conn(0x7f5adc15cb80 legacy=0x7f5adc152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:29.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ad3e00640 1 -- 172.21.15.81:0/3118143551 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc0c52f0 con 0x7f5adc06c790 2024-05-27T06:56:29.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.975+0000 7f5ae1600640 1 --2- 172.21.15.81:0/3118143551 >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 0x7f5adc0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:56:29.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/3118143551 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc06cba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40097c0 tx=0x7f5ac401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb0299e263987f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:29.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40b0070 con 0x7f5adc06c790 2024-05-27T06:56:29.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac40ab400 con 0x7f5adc06c790 2024-05-27T06:56:29.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5adbe00640 1 -- 172.21.15.81:0/3118143551 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5acc001720 con 0x7f5adc06c790 2024-05-27T06:56:29.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 -- 172.21.15.81:0/3118143551 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 msgr2=0x7f5adc06cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 --2- 172.21.15.81:0/3118143551 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc06cba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40097c0 tx=0x7f5ac401c3c0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 -- 172.21.15.81:0/3118143551 shutdown_connections 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 --2- 172.21.15.81:0/3118143551 >> v2:172.21.15.106:3300/0 conn(0x7f5adc0c8880 0x7f5adc0c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 --2- 172.21.15.81:0/3118143551 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc06cba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 -- 172.21.15.81:0/3118143551 >> 172.21.15.81:0/3118143551 conn(0x7f5adc070700 msgr2=0x7f5adc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 -- 172.21.15.81:0/3118143551 shutdown_connections 2024-05-27T06:56:29.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 -- 172.21.15.81:0/3118143551 wait complete. 2024-05-27T06:56:29.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.979+0000 7f5ae3600640 1 Processor -- start 2024-05-27T06:56:29.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- start start 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 0x7f5adc14b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc0c9c80 con 0x7f5adc0c8880 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc0c9b00 con 0x7f5adc15cb80 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc1654f0 con 0x7f5adc06c790 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49694/0 (socket says 172.21.15.81:49694) 2024-05-27T06:56:29.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 -- 172.21.15.81:0/4278881308 learned_addr learned my addr 172.21.15.81:0/4278881308 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae1600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 0x7f5adc14b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae0c00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 -- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 msgr2=0x7f5adc1f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc1f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 -- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 msgr2=0x7f5adc14b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 0x7f5adc14b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 -- 172.21.15.81:0/4278881308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc0cc290 con 0x7f5adc0c8880 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae1600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 0x7f5adc14b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:29.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad3e00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5ac401c8a0 tx=0x7f5ac4019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40b0040 con 0x7f5adc0c8880 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac4003620 con 0x7f5adc0c8880 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5adc0cdd60 con 0x7f5adc0c8880 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40039d0 con 0x7f5adc0c8880 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5adc145790 con 0x7f5adc0c8880 2024-05-27T06:56:29.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.983+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5adc06a560 con 0x7f5adc0c8880 2024-05-27T06:56:29.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.987+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5ac40bd020 con 0x7f5adc0c8880 2024-05-27T06:56:29.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.987+0000 7f5ad9600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 0x7f5ab8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:29.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.987+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f5ac4128f00 con 0x7f5adc0c8880 2024-05-27T06:56:29.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.987+0000 7f5ae0c00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 0x7f5ab8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:29.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.987+0000 7f5ae0c00640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 0x7f5ab8071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5acc001250 tx=0x7f5acc001c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:29.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:29.991+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5ac4015fb0 con 0x7f5adc0c8880 2024-05-27T06:56:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.183+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5adc151230 con 0x7f5adc0c8880 2024-05-27T06:56:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.183+0000 7f5ad9600640 1 -- 172.21.15.81:0/4278881308 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f5ac40f5b70 con 0x7f5adc0c8880 2024-05-27T06:56:30.175 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:30.175 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:30.175 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 msgr2=0x7f5ab8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 0x7f5ab8071910 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5acc001250 tx=0x7f5acc001c60 comp rx=0 tx=0).stop 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 msgr2=0x7f5adc0cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5ac401c8a0 tx=0x7f5ac4019160 comp rx=0 tx=0).stop 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 shutdown_connections 2024-05-27T06:56:30.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5ab806f400 0x7f5ab8071910 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5adc15cb80 0x7f5adc14b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5adc0c8880 0x7f5adc0cbd50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 --2- 172.21.15.81:0/4278881308 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5adc06c790 0x7f5adc1f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 >> 172.21.15.81:0/4278881308 conn(0x7f5adc070700 msgr2=0x7f5adc150570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 shutdown_connections 2024-05-27T06:56:30.178 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:30.187+0000 7f5ae3600640 1 -- 172.21.15.81:0/4278881308 wait complete. 2024-05-27T06:56:30.197 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:56:31.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~27s 2024-05-27T06:56:32.198 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:32.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 Processor -- start 2024-05-27T06:56:32.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 -- start start 2024-05-27T06:56:32.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9fe81546f0 0x7f9fe8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 0x7f9fe80c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fe815db00 con 0x7f9fe80c7510 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe815e6f0 con 0x7f9fe81546f0 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.367+0000 7f9ff0200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe80755b0 con 0x7f9fe80732f0 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fee200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9fe80c7510 0x7f9fe815a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50860/0 (socket says 172.21.15.81:50860) 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fee200640 1 -- 172.21.15.81:0/598344376 learned_addr learned my addr 172.21.15.81:0/598344376 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 --2- 172.21.15.81:0/598344376 >> v2:172.21.15.81:3300/0 conn(0x7f9fe81546f0 0x7f9fe8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fece00640 1 --2- 172.21.15.81:0/598344376 >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 0x7f9fe80c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 -- 172.21.15.81:0/598344376 >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 msgr2=0x7f9fe80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 --2- 172.21.15.81:0/598344376 >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 0x7f9fe80c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 -- 172.21.15.81:0/598344376 >> v1:172.21.15.81:6789/0 conn(0x7f9fe80c7510 legacy=0x7f9fe815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 -- 172.21.15.81:0/598344376 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe8073860 con 0x7f9fe81546f0 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fe7e00640 1 -- 172.21.15.81:0/598344376 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432458605 0 0) 0x7f9fe815db00 con 0x7f9fe80c7510 2024-05-27T06:56:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fece00640 1 --2- 172.21.15.81:0/598344376 >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 0x7f9fe80c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:32.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fed800640 1 --2- 172.21.15.81:0/598344376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe81546f0 0x7f9fe8072db0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9fe806da10 tx=0x7f9fdc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e966c71d10e524 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.371+0000 7f9fe7e00640 1 -- 172.21.15.81:0/598344376 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0b5070 con 0x7f9fe81546f0 2024-05-27T06:56:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9fe7e00640 1 -- 172.21.15.81:0/598344376 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fdc0a8970 con 0x7f9fe81546f0 2024-05-27T06:56:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9fe7e00640 1 -- 172.21.15.81:0/598344376 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0a8d40 con 0x7f9fe81546f0 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- 172.21.15.81:0/598344376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe81546f0 msgr2=0x7f9fe8072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 --2- 172.21.15.81:0/598344376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe81546f0 0x7f9fe8072db0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9fe806da10 tx=0x7f9fdc01ec70 comp rx=0 tx=0).stop 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- 172.21.15.81:0/598344376 shutdown_connections 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 --2- 172.21.15.81:0/598344376 >> v2:172.21.15.106:3300/0 conn(0x7f9fe80732f0 0x7f9fe80c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 --2- 172.21.15.81:0/598344376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe81546f0 0x7f9fe8072db0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- 172.21.15.81:0/598344376 >> 172.21.15.81:0/598344376 conn(0x7f9fe8070700 msgr2=0x7f9fe80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- 172.21.15.81:0/598344376 shutdown_connections 2024-05-27T06:56:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- 172.21.15.81:0/598344376 wait complete. 2024-05-27T06:56:32.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 Processor -- start 2024-05-27T06:56:32.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.375+0000 7f9ff0200640 1 -- start start 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9fe80732f0 0x7f9fe8150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe81546f0 0x7f9fe815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe815e380 con 0x7f9fe80c7510 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe815e0a0 con 0x7f9fe81546f0 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe8154b00 con 0x7f9fe80732f0 2024-05-27T06:56:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49714/0 (socket says 172.21.15.81:49714) 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 -- 172.21.15.81:0/2696081515 learned_addr learned my addr 172.21.15.81:0/2696081515 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fed800640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9fe80732f0 0x7f9fe8150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fee200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe81546f0 0x7f9fe815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 -- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9fe80732f0 msgr2=0x7f9fe8150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9fe80732f0 0x7f9fe8150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 -- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe81546f0 msgr2=0x7f9fe815a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe81546f0 0x7f9fe815a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 -- 172.21.15.81:0/2696081515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe815ad80 con 0x7f9fe80c7510 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fece00640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400d7d0 tx=0x7f9fd400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400b6c0 con 0x7f9fe80c7510 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fd4019070 con 0x7f9fe80c7510 2024-05-27T06:56:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe8153e10 con 0x7f9fe80c7510 2024-05-27T06:56:32.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400ba60 con 0x7f9fe80c7510 2024-05-27T06:56:32.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81f2f20 con 0x7f9fe80c7510 2024-05-27T06:56:32.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.379+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe806a560 con 0x7f9fe80c7510 2024-05-27T06:56:32.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.383+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9fd400bc00 con 0x7f9fe80c7510 2024-05-27T06:56:32.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.383+0000 7f9fe5600640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 0x7f9fc8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:32.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.383+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f9fd408f970 con 0x7f9fe80c7510 2024-05-27T06:56:32.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.383+0000 7f9fed800640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 0x7f9fc8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:32.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.383+0000 7f9fed800640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 0x7f9fc8071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0b2000 tx=0x7f9fdc0aa070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:32.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.387+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9fd405c540 con 0x7f9fe80c7510 2024-05-27T06:56:32.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.579+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fe815db00 con 0x7f9fe80c7510 2024-05-27T06:56:32.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.579+0000 7f9fe5600640 1 -- 172.21.15.81:0/2696081515 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f9fd405be40 con 0x7f9fe80c7510 2024-05-27T06:56:32.571 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:32.572 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:32.572 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 msgr2=0x7f9fc8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 0x7f9fc8071910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0b2000 tx=0x7f9fdc0aa070 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 msgr2=0x7f9fe8158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400d7d0 tx=0x7f9fd400dc90 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 shutdown_connections 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9fc806f400 0x7f9fc8071910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe81546f0 0x7f9fe815a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fe80c7510 0x7f9fe8158d90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 --2- 172.21.15.81:0/2696081515 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9fe80732f0 0x7f9fe8150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 >> 172.21.15.81:0/2696081515 conn(0x7f9fe8070700 msgr2=0x7f9fe81515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:32.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 shutdown_connections 2024-05-27T06:56:32.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:32.583+0000 7f9ff0200640 1 -- 172.21.15.81:0/2696081515 wait complete. 2024-05-27T06:56:32.593 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:34.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:56:34.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:56:34.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:34.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.775+0000 7f19c0c00640 1 Processor -- start 2024-05-27T06:56:34.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.775+0000 7f19c0c00640 1 -- start start 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19c0c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f19bc15cb80 0x7f19bc152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19c0c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 0x7f19bc06cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19c0c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19bc0c9f80 con 0x7f19bc0c8880 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19c0c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc0c58b0 con 0x7f19bc15cb80 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19c0c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f19bc0c83f0 con 0x7f19bc06c790 2024-05-27T06:56:34.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f19bc15cb80 0x7f19bc152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f19bc15cb80 0x7f19bc152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49730/0 (socket says 172.21.15.81:49730) 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 -- 172.21.15.81:0/1817075787 learned_addr learned my addr 172.21.15.81:0/1817075787 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9400640 1 --2- 172.21.15.81:0/1817075787 >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 0x7f19bc06cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 -- 172.21.15.81:0/1817075787 >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 msgr2=0x7f19bc06cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 --2- 172.21.15.81:0/1817075787 >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 0x7f19bc06cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 -- 172.21.15.81:0/1817075787 >> v1:172.21.15.106:6789/0 conn(0x7f19bc0c8880 legacy=0x7f19bc0c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9e00640 1 -- 172.21.15.81:0/1817075787 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc0c52f0 con 0x7f19bc15cb80 2024-05-27T06:56:34.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.779+0000 7f19b9400640 1 --2- 172.21.15.81:0/1817075787 >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 0x7f19bc06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:34.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19b9e00640 1 --2- 172.21.15.81:0/1817075787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc15cb80 0x7f19bc152cb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f19bc071360 tx=0x7f19ac01d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2504fea0425b98 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19b8a00640 1 -- 172.21.15.81:0/1817075787 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac0b3070 con 0x7f19bc15cb80 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19b8a00640 1 -- 172.21.15.81:0/1817075787 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19ac0ae680 con 0x7f19bc15cb80 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19b8a00640 1 -- 172.21.15.81:0/1817075787 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac0aea50 con 0x7f19bc15cb80 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 -- 172.21.15.81:0/1817075787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc15cb80 msgr2=0x7f19bc152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 --2- 172.21.15.81:0/1817075787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc15cb80 0x7f19bc152cb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f19bc071360 tx=0x7f19ac01d630 comp rx=0 tx=0).stop 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 -- 172.21.15.81:0/1817075787 shutdown_connections 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 --2- 172.21.15.81:0/1817075787 >> v2:172.21.15.106:3300/0 conn(0x7f19bc06c790 0x7f19bc06cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 --2- 172.21.15.81:0/1817075787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc15cb80 0x7f19bc152cb0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 -- 172.21.15.81:0/1817075787 >> 172.21.15.81:0/1817075787 conn(0x7f19bc070700 msgr2=0x7f19bc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 -- 172.21.15.81:0/1817075787 shutdown_connections 2024-05-27T06:56:34.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.783+0000 7f19c0c00640 1 -- 172.21.15.81:0/1817075787 wait complete. 2024-05-27T06:56:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 Processor -- start 2024-05-27T06:56:34.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- start start 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19bc06c790 0x7f19bc1f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19bc15cb80 0x7f19bc159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc0c9c80 con 0x7f19bc0c8880 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f19bc0c9b00 con 0x7f19bc06c790 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f19bc167700 con 0x7f19bc15cb80 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49732/0 (socket says 172.21.15.81:49732) 2024-05-27T06:56:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 -- 172.21.15.81:0/2576317487 learned_addr learned my addr 172.21.15.81:0/2576317487 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9e00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19bc06c790 0x7f19bc1f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19ba800640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19bc15cb80 0x7f19bc159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 -- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19bc15cb80 msgr2=0x7f19bc159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19bc15cb80 0x7f19bc159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 -- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19bc06c790 msgr2=0x7f19bc1f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19bc06c790 0x7f19bc1f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 -- 172.21.15.81:0/2576317487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19bc1f3d90 con 0x7f19bc0c8880 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b9400640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f19a800c7f0 tx=0x7f19a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8015070 con 0x7f19bc0c8880 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a8004220 con 0x7f19bc0c8880 2024-05-27T06:56:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19bc0745e0 con 0x7f19bc0c8880 2024-05-27T06:56:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8003620 con 0x7f19bc0c8880 2024-05-27T06:56:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19bc1f7080 con 0x7f19bc0c8880 2024-05-27T06:56:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.787+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19bc06a560 con 0x7f19bc0c8880 2024-05-27T06:56:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.791+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f19a80039d0 con 0x7f19bc0c8880 2024-05-27T06:56:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.791+0000 7f19b2000640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.791+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f19a808f3c0 con 0x7f19bc0c8880 2024-05-27T06:56:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.791+0000 7f19b9e00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:34.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.791+0000 7f19b9e00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 0x7f199c071940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0b0000 tx=0x7f19ac0a8000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.795+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f19a805bb00 con 0x7f19bc0c8880 2024-05-27T06:56:34.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.987+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19bc0c8ee0 con 0x7f19bc0c8880 2024-05-27T06:56:34.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.987+0000 7f19b2000640 1 -- 172.21.15.81:0/2576317487 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f19a805fe00 con 0x7f19bc0c8880 2024-05-27T06:56:34.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:34.978 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:34.979 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:34.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 msgr2=0x7f199c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 0x7f199c071940 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19ac0b0000 tx=0x7f19ac0a8000 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 msgr2=0x7f19bc149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f19a800c7f0 tx=0x7f19a800ccb0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 shutdown_connections 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19bc15cb80 0x7f19bc159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f199c06f430 0x7f199c071940 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19bc0c8880 0x7f19bc149280 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 --2- 172.21.15.81:0/2576317487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f19bc06c790 0x7f19bc1f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:34.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 >> 172.21.15.81:0/2576317487 conn(0x7f19bc070700 msgr2=0x7f19bc0c6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:34.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 shutdown_connections 2024-05-27T06:56:34.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:34.991+0000 7f19c0c00640 1 -- 172.21.15.81:0/2576317487 wait complete. 2024-05-27T06:56:35.000 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:37.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:37.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.167+0000 7fe8a5600640 1 Processor -- start 2024-05-27T06:56:37.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe8a5600640 1 -- start start 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe8a006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe8a5600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a00c9f80 con 0x7fe8a00c8880 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe8a5600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a00c52f0 con 0x7fe8a015cb80 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe8a5600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a00c58b0 con 0x7fe8a006c790 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89f400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe8a00c8880 0x7fe8a00c8c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47642/0 (socket says 172.21.15.81:47642) 2024-05-27T06:56:37.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89f400640 1 -- 172.21.15.81:0/2311505602 learned_addr learned my addr 172.21.15.81:0/2311505602 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:37.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3311406449 0 0) 0x7fe8a00c9f80 con 0x7fe8a00c8880 2024-05-27T06:56:37.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe884003530 con 0x7fe8a00c8880 2024-05-27T06:56:37.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe896000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe8a006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3964025909 0 0) 0x7fe884003530 con 0x7fe8a00c8880 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 msgr2=0x7fe8a006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe8a006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 >> v1:172.21.15.106:6789/0 conn(0x7fe8a015cb80 legacy=0x7fe8a0152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a00c55d0 con 0x7fe8a00c8880 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fe898001720 con 0x7fe8a00c8880 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe896000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe8a006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 >> v1:172.21.15.81:6789/0 conn(0x7fe8a00c8880 legacy=0x7fe8a00c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe884003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe884004810 0x7fe884004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe88400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a00c58b0 con 0x7fe884003a40 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe884005560 con 0x7fe884004810 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe88400d020 con 0x7fe8a006c790 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe884003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe896000640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe884004810 0x7fe884004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89f400640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe88400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 -- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 msgr2=0x7fe88400cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:37.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a006c790 0x7fe88400cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 -- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe884004810 msgr2=0x7fe884004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:37.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe884004810 0x7fe884004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.171+0000 7fe89ea00640 1 -- 172.21.15.81:0/2311505602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe884003530 con 0x7fe884003a40 2024-05-27T06:56:37.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe89ea00640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe884003eb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe890001e90 tx=0x7fe89001c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5149511407046e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:37.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900b0070 con 0x7fe884003a40 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe890003b80 con 0x7fe884003a40 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe89e000640 1 -- 172.21.15.81:0/2311505602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890001ae0 con 0x7fe884003a40 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 -- 172.21.15.81:0/2311505602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 msgr2=0x7fe884003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 --2- 172.21.15.81:0/2311505602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe884003eb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe890001e90 tx=0x7fe89001c770 comp rx=0 tx=0).stop 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe89f400640 1 -- 172.21.15.81:0/2311505602 reap_dead start 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 -- 172.21.15.81:0/2311505602 shutdown_connections 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 -- 172.21.15.81:0/2311505602 >> 172.21.15.81:0/2311505602 conn(0x7fe8a0070700 msgr2=0x7fe8a00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 -- 172.21.15.81:0/2311505602 shutdown_connections 2024-05-27T06:56:37.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.175+0000 7fe8a5600640 1 -- 172.21.15.81:0/2311505602 wait complete. 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 Processor -- start 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 -- start start 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 0x7fe8a0146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a014b800 0x7fe8a014bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a00c83f0 con 0x7fe884003a40 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a01644f0 con 0x7fe8a0159d20 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe8a5600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a0164670 con 0x7fe8a014b800 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe89ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe89ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56478/0 (socket says 172.21.15.81:56478) 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.179+0000 7fe89ea00640 1 -- 172.21.15.81:0/4006339027 learned_addr learned my addr 172.21.15.81:0/4006339027 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe896000640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 0x7fe8a0146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89f400640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a014b800 0x7fe8a014bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 -- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a014b800 msgr2=0x7fe8a014bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a014b800 0x7fe8a014bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 -- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 msgr2=0x7fe8a0146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 0x7fe8a0146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 -- 172.21.15.81:0/4006339027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a01f9700 con 0x7fe884003a40 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe896000640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 0x7fe8a0146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe89ea00640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe890005040 tx=0x7fe884005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:37.174 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900b0040 con 0x7fe884003a40 2024-05-27T06:56:37.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8900be070 con 0x7fe884003a40 2024-05-27T06:56:37.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a0156e40 con 0x7fe884003a40 2024-05-27T06:56:37.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900a7070 con 0x7fe884003a40 2024-05-27T06:56:37.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a014a060 con 0x7fe884003a40 2024-05-27T06:56:37.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a006a560 con 0x7fe884003a40 2024-05-27T06:56:37.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.183+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe8900ab470 con 0x7fe884003a40 2024-05-27T06:56:37.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.187+0000 7fe897400640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 0x7fe87c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:37.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.187+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fe890129880 con 0x7fe884003a40 2024-05-27T06:56:37.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.187+0000 7fe896000640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 0x7fe87c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:37.177 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.187+0000 7fe896000640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 0x7fe87c071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0149aa0 tx=0x7fe888008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:37.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.191+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe8900164d0 con 0x7fe884003a40 2024-05-27T06:56:37.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.379+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8a014c150 con 0x7fe884003a40 2024-05-27T06:56:37.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.379+0000 7fe897400640 1 -- 172.21.15.81:0/4006339027 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fe8a014c150 con 0x7fe884003a40 2024-05-27T06:56:37.371 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:37.372 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:37.372 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 msgr2=0x7fe87c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 0x7fe87c071910 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0149aa0 tx=0x7fe888008040 comp rx=0 tx=0).stop 2024-05-27T06:56:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 msgr2=0x7fe8a01f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe890005040 tx=0x7fe884005840 comp rx=0 tx=0).stop 2024-05-27T06:56:37.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 shutdown_connections 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe8a014b800 0x7fe8a014bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe87c06f400 0x7fe87c071910 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8a0159d20 0x7fe8a0146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 --2- 172.21.15.81:0/4006339027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe884003a40 0x7fe8a01f91c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 >> 172.21.15.81:0/4006339027 conn(0x7fe8a0070700 msgr2=0x7fe8a00cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 shutdown_connections 2024-05-27T06:56:37.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:37.383+0000 7fe8a5600640 1 -- 172.21.15.81:0/4006339027 wait complete. 2024-05-27T06:56:37.393 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:37.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~34s 2024-05-27T06:56:39.394 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:39.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 Processor -- start 2024-05-27T06:56:39.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 -- start start 2024-05-27T06:56:39.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c15db00 con 0x7f463c0732f0 2024-05-27T06:56:39.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f463c15e6f0 con 0x7f463c15a510 2024-05-27T06:56:39.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.563+0000 7f4643600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f463c0755b0 con 0x7f463c1546f0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4641600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:46354/0 (socket says 172.21.15.81:46354) 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4641600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:54060/0 (socket says 172.21.15.81:54060) 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4641600640 1 -- 172.21.15.81:0/1819667287 learned_addr learned my addr 172.21.15.81:0/1819667287 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f463be00640 1 -- 172.21.15.81:0/1819667287 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064561112 0 0) 0x7f463c15db00 con 0x7f463c0732f0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f463be00640 1 -- 172.21.15.81:0/1819667287 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4620003530 con 0x7f463c0732f0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 -- 172.21.15.81:0/1819667287 >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 msgr2=0x7f463c15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 --2- 172.21.15.81:0/1819667287 >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 -- 172.21.15.81:0/1819667287 >> v1:172.21.15.106:6789/0 conn(0x7f463c0732f0 legacy=0x7f463c0c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 -- 172.21.15.81:0/1819667287 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c15ae60 con 0x7f463c1546f0 2024-05-27T06:56:39.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4641600640 1 --2- 172.21.15.81:0/1819667287 >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:56:39.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.567+0000 7f4640c00640 1 --2- 172.21.15.81:0/1819667287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097c0 tx=0x7f462c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff3df20ffaca0093 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:39.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f463be00640 1 -- 172.21.15.81:0/1819667287 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0b5070 con 0x7f463c1546f0 2024-05-27T06:56:39.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f463be00640 1 -- 172.21.15.81:0/1819667287 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c0a8990 con 0x7f463c1546f0 2024-05-27T06:56:39.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f463be00640 1 -- 172.21.15.81:0/1819667287 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0a8d40 con 0x7f463c1546f0 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- 172.21.15.81:0/1819667287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 msgr2=0x7f463c072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 --2- 172.21.15.81:0/1819667287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097c0 tx=0x7f462c01ec90 comp rx=0 tx=0).stop 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- 172.21.15.81:0/1819667287 shutdown_connections 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 --2- 172.21.15.81:0/1819667287 >> v2:172.21.15.106:3300/0 conn(0x7f463c15a510 0x7f463c15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 --2- 172.21.15.81:0/1819667287 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c072db0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- 172.21.15.81:0/1819667287 >> 172.21.15.81:0/1819667287 conn(0x7f463c070700 msgr2=0x7f463c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- 172.21.15.81:0/1819667287 shutdown_connections 2024-05-27T06:56:39.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- 172.21.15.81:0/1819667287 wait complete. 2024-05-27T06:56:39.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 Processor -- start 2024-05-27T06:56:39.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.571+0000 7f4643600640 1 -- start start 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c0cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f463c15a510 0x7f463c0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f463c15e0a0 con 0x7f463c0732f0 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f463c15ddc0 con 0x7f463c15a510 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f463c15e380 con 0x7f463c1546f0 2024-05-27T06:56:39.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56490/0 (socket says 172.21.15.81:56490) 2024-05-27T06:56:39.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 -- 172.21.15.81:0/794007297 learned_addr learned my addr 172.21.15.81:0/794007297 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:39.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4641600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f463c15a510 0x7f463c0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4633e00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c0cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 -- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 msgr2=0x7f463c0cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c0cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 -- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f463c15a510 msgr2=0x7f463c0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f463c15a510 0x7f463c0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 -- 172.21.15.81:0/794007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c1f3ac0 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4640c00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097c0 tx=0x7f462c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0c0070 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c004230 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c1f7060 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0b5040 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c157ff0 con 0x7f463c0732f0 2024-05-27T06:56:39.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.575+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c1f3c90 con 0x7f463c0732f0 2024-05-27T06:56:39.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.579+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f462c0c3070 con 0x7f463c0732f0 2024-05-27T06:56:39.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.579+0000 7f4639600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 0x7f4618071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:39.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.579+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f462c12bbe0 con 0x7f463c0732f0 2024-05-27T06:56:39.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.579+0000 7f4633e00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 0x7f4618071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:39.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.579+0000 7f4633e00640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 0x7f4618071940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f463c0c8b80 tx=0x7f4624009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:39.574 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.583+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f462c0187f0 con 0x7f463c0732f0 2024-05-27T06:56:39.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.775+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f463c06a560 con 0x7f463c0732f0 2024-05-27T06:56:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.775+0000 7f4639600640 1 -- 172.21.15.81:0/794007297 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f462c0f8160 con 0x7f463c0732f0 2024-05-27T06:56:39.767 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:39.767 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 msgr2=0x7f4618071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 0x7f4618071940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f463c0c8b80 tx=0x7f4624009040 comp rx=0 tx=0).stop 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 msgr2=0x7f463c1f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097c0 tx=0x7f462c01ba30 comp rx=0 tx=0).stop 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 shutdown_connections 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f463c1546f0 0x7f463c0cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f461806f430 0x7f4618071940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f463c15a510 0x7f463c0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 --2- 172.21.15.81:0/794007297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f463c0732f0 0x7f463c1f3580 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 >> 172.21.15.81:0/794007297 conn(0x7f463c070700 msgr2=0x7f463c159720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 shutdown_connections 2024-05-27T06:56:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:39.779+0000 7f4643600640 1 -- 172.21.15.81:0/794007297 wait complete. 2024-05-27T06:56:39.789 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:41.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:41.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.959+0000 7f6ef4a00640 1 Processor -- start 2024-05-27T06:56:41.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.959+0000 7f6ef4a00640 1 -- start start 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef0152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ef4a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 0x7f6ef006c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ef4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef00c9f80 con 0x7f6ef00ce3e0 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ef4a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef00c58b0 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00c83f0 con 0x7f6ef015cb80 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eee800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6ef00ce3e0 0x7f6ef00c8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47646/0 (socket says 172.21.15.81:47646) 2024-05-27T06:56:41.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eee800640 1 -- 172.21.15.81:0/3902147136 learned_addr learned my addr 172.21.15.81:0/3902147136 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:41.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eede00640 1 --2- 172.21.15.81:0/3902147136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef0152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ee5400640 1 --2- 172.21.15.81:0/3902147136 >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 0x7f6ef006c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eed400640 1 -- 172.21.15.81:0/3902147136 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2769216568 0 0) 0x7f6ef00c9f80 con 0x7f6ef00ce3e0 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eed400640 1 -- 172.21.15.81:0/3902147136 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ed4003530 con 0x7f6ef00ce3e0 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eede00640 1 -- 172.21.15.81:0/3902147136 >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 msgr2=0x7f6ef006c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eede00640 1 --2- 172.21.15.81:0/3902147136 >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 0x7f6ef006c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eede00640 1 -- 172.21.15.81:0/3902147136 >> v1:172.21.15.81:6789/0 conn(0x7f6ef00ce3e0 legacy=0x7f6ef00c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6eede00640 1 -- 172.21.15.81:0/3902147136 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef00c52f0 con 0x7f6ef015cb80 2024-05-27T06:56:41.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.963+0000 7f6ee5400640 1 --2- 172.21.15.81:0/3902147136 >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 0x7f6ef006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:56:41.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6eede00640 1 --2- 172.21.15.81:0/3902147136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef0152cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009d90 tx=0x7f6ee001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc44fff83b13d5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6eed400640 1 -- 172.21.15.81:0/3902147136 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00b2070 con 0x7f6ef015cb80 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6eed400640 1 -- 172.21.15.81:0/3902147136 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee00a58c0 con 0x7f6ef015cb80 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6eed400640 1 -- 172.21.15.81:0/3902147136 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00a5c90 con 0x7f6ef015cb80 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 -- 172.21.15.81:0/3902147136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 msgr2=0x7f6ef0152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/3902147136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef0152cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009d90 tx=0x7f6ee001c970 comp rx=0 tx=0).stop 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 -- 172.21.15.81:0/3902147136 shutdown_connections 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/3902147136 >> v2:172.21.15.106:3300/0 conn(0x7f6ef00cdfd0 0x7f6ef006c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/3902147136 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef0152cb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 -- 172.21.15.81:0/3902147136 >> 172.21.15.81:0/3902147136 conn(0x7f6ef0070700 msgr2=0x7f6ef00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 -- 172.21.15.81:0/3902147136 shutdown_connections 2024-05-27T06:56:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.967+0000 7f6ef4a00640 1 -- 172.21.15.81:0/3902147136 wait complete. 2024-05-27T06:56:41.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 Processor -- start 2024-05-27T06:56:41.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- start start 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ef00ce3e0 0x7f6ef0162980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef00c9c80 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef00c9b00 con 0x7f6ef00ce3e0 2024-05-27T06:56:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef015d4b0 con 0x7f6ef015cb80 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56504/0 (socket says 172.21.15.81:56504) 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 -- 172.21.15.81:0/1943179720 learned_addr learned my addr 172.21.15.81:0/1943179720 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eee800640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ee5400640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ef00ce3e0 0x7f6ef0162980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 -- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 msgr2=0x7f6ef014bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef014bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 -- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ef00ce3e0 msgr2=0x7f6ef0162980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ef00ce3e0 0x7f6ef0162980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 -- 172.21.15.81:0/1943179720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef0162ec0 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6eede00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001a5a0 tx=0x7f6ee0019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00b2070 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee00b3040 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00b9690 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef014c1d0 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.971+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef01f6b30 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef006a560 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6ee00c1030 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ee6a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 0x7f6ecc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:41.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6ee0129ca0 con 0x7f6ef00cdfd0 2024-05-27T06:56:41.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ee5400640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 0x7f6ecc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:41.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.975+0000 7f6ee5400640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 0x7f6ecc071940 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ef014c900 tx=0x7f6ed8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:41.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:41.979+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6ee00c8060 con 0x7f6ef00cdfd0 2024-05-27T06:56:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.171+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ef0072b40 con 0x7f6ef00cdfd0 2024-05-27T06:56:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.171+0000 7f6ee6a00640 1 -- 172.21.15.81:0/1943179720 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6ef0072b40 con 0x7f6ef00cdfd0 2024-05-27T06:56:42.163 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:42.163 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:42.163 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 msgr2=0x7f6ecc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 0x7f6ecc071940 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6ef014c900 tx=0x7f6ed8008040 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 msgr2=0x7f6ef0159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001a5a0 tx=0x7f6ee0019710 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 shutdown_connections 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ef015cb80 0x7f6ef014bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6ecc06f430 0x7f6ecc071940 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ef00ce3e0 0x7f6ef0162980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 --2- 172.21.15.81:0/1943179720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ef00cdfd0 0x7f6ef0159710 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 >> 172.21.15.81:0/1943179720 conn(0x7f6ef0070700 msgr2=0x7f6ef00cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 shutdown_connections 2024-05-27T06:56:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:42.175+0000 7f6ef4a00640 1 -- 172.21.15.81:0/1943179720 wait complete. 2024-05-27T06:56:42.185 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:56:44.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.355+0000 7f3c00200640 1 Processor -- start 2024-05-27T06:56:44.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.355+0000 7f3c00200640 1 -- start start 2024-05-27T06:56:44.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3c00200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 0x7f3bf8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3c00200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3bf80732f0 0x7f3bf80c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3c00200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf815db00 con 0x7f3bf80c7510 2024-05-27T06:56:44.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3c00200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3bf815e6f0 con 0x7f3bf80732f0 2024-05-27T06:56:44.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3c00200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f3bf80755b0 con 0x7f3bf81546f0 2024-05-27T06:56:44.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3bf80732f0 0x7f3bf80c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3bf80732f0 0x7f3bf80c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56516/0 (socket says 172.21.15.81:56516) 2024-05-27T06:56:44.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 -- 172.21.15.81:0/2734852305 learned_addr learned my addr 172.21.15.81:0/2734852305 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:44.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfd800640 1 --2- 172.21.15.81:0/2734852305 >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 0x7f3bf8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 -- 172.21.15.81:0/2734852305 >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 msgr2=0x7f3bf8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:44.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 --2- 172.21.15.81:0/2734852305 >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 0x7f3bf8072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 -- 172.21.15.81:0/2734852305 >> v1:172.21.15.106:6789/0 conn(0x7f3bf80c7510 legacy=0x7f3bf815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:44.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 -- 172.21.15.81:0/2734852305 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf8073860 con 0x7f3bf80732f0 2024-05-27T06:56:44.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfd800640 1 --2- 172.21.15.81:0/2734852305 >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 0x7f3bf8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:44.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.359+0000 7f3bfce00640 1 --2- 172.21.15.81:0/2734852305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf80c6fd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3bf806d950 tx=0x7f3be401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d871ac8dbc2bf98e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:44.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3bf7e00640 1 -- 172.21.15.81:0/2734852305 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be40b5070 con 0x7f3bf80732f0 2024-05-27T06:56:44.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3bf7e00640 1 -- 172.21.15.81:0/2734852305 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be40a89d0 con 0x7f3bf80732f0 2024-05-27T06:56:44.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3bf7e00640 1 -- 172.21.15.81:0/2734852305 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be40a8da0 con 0x7f3bf80732f0 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- 172.21.15.81:0/2734852305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 msgr2=0x7f3bf80c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 --2- 172.21.15.81:0/2734852305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf80c6fd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3bf806d950 tx=0x7f3be401ed10 comp rx=0 tx=0).stop 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- 172.21.15.81:0/2734852305 shutdown_connections 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 --2- 172.21.15.81:0/2734852305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf80c6fd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 --2- 172.21.15.81:0/2734852305 >> v2:172.21.15.106:3300/0 conn(0x7f3bf81546f0 0x7f3bf8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- 172.21.15.81:0/2734852305 >> 172.21.15.81:0/2734852305 conn(0x7f3bf8070700 msgr2=0x7f3bf80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- 172.21.15.81:0/2734852305 shutdown_connections 2024-05-27T06:56:44.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- 172.21.15.81:0/2734852305 wait complete. 2024-05-27T06:56:44.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 Processor -- start 2024-05-27T06:56:44.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.363+0000 7f3c00200640 1 -- start start 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf80c7510 0x7f3bf81574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 0x7f3bf8157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf815e380 con 0x7f3bf80732f0 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf815e0a0 con 0x7f3bf81546f0 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf8154b00 con 0x7f3bf80c7510 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56530/0 (socket says 172.21.15.81:56530) 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 -- 172.21.15.81:0/113672742 learned_addr learned my addr 172.21.15.81:0/113672742 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfce00640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf80c7510 0x7f3bf81574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfe200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 0x7f3bf8157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 -- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf80c7510 msgr2=0x7f3bf81574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf80c7510 0x7f3bf81574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 -- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 msgr2=0x7f3bf8157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 0x7f3bf8157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 -- 172.21.15.81:0/113672742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf81f3830 con 0x7f3bf80732f0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfe200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 0x7f3bf8157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bfd800640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00ce60 tx=0x7f3bec00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00f890 con 0x7f3bf80732f0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bec00fa30 con 0x7f3bf80732f0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf8159d00 con 0x7f3bf80732f0 2024-05-27T06:56:44.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec00fe00 con 0x7f3bf80732f0 2024-05-27T06:56:44.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf80c4ef0 con 0x7f3bf80732f0 2024-05-27T06:56:44.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.367+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf806a560 con 0x7f3bf80732f0 2024-05-27T06:56:44.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.371+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3bec018400 con 0x7f3bf80732f0 2024-05-27T06:56:44.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.371+0000 7f3bf5600640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 0x7f3bd8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:44.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.371+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f3bec0902e0 con 0x7f3bf80732f0 2024-05-27T06:56:44.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.371+0000 7f3bfce00640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 0x7f3bd8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:44.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.371+0000 7f3bfce00640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 0x7f3bd8071940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b2000 tx=0x7f3be4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:44.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.375+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3bec014030 con 0x7f3bf80732f0 2024-05-27T06:56:44.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bf81f2c30 con 0x7f3bf80732f0 2024-05-27T06:56:44.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3bf5600640 1 -- 172.21.15.81:0/113672742 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f3bec00ae20 con 0x7f3bf80732f0 2024-05-27T06:56:44.557 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:44.557 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:44.558 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 msgr2=0x7f3bd8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 0x7f3bd8071940 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b2000 tx=0x7f3be4008040 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 msgr2=0x7f3bf81f32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00ce60 tx=0x7f3bec00a650 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 shutdown_connections 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3bf80c7510 0x7f3bf81574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3bd806f430 0x7f3bd8071940 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bf81546f0 0x7f3bf8157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 --2- 172.21.15.81:0/113672742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bf80732f0 0x7f3bf81f32f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 >> 172.21.15.81:0/113672742 conn(0x7f3bf8070700 msgr2=0x7f3bf80c9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 shutdown_connections 2024-05-27T06:56:44.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:44.567+0000 7f3c00200640 1 -- 172.21.15.81:0/113672742 wait complete. 2024-05-27T06:56:44.580 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:44.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~40s 2024-05-27T06:56:46.581 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:46.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.751+0000 7ff411400640 1 Processor -- start 2024-05-27T06:56:46.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 -- start start 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff40c1546f0 0x7ff40c072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff40c15a510 0x7ff40c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff40c15db00 con 0x7ff40c0732f0 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c15e6f0 con 0x7ff40c15a510 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff411400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c0755b0 con 0x7ff40c1546f0 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff409e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff40c0732f0 0x7ff40c0c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44522/0 (socket says 172.21.15.81:44522) 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff409e00640 1 -- 172.21.15.81:0/4282533918 learned_addr learned my addr 172.21.15.81:0/4282533918 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40b200640 1 --2- 172.21.15.81:0/4282533918 >> v2:172.21.15.81:3300/0 conn(0x7ff40c15a510 0x7ff40c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40a800640 1 --2- 172.21.15.81:0/4282533918 >> v2:172.21.15.106:3300/0 conn(0x7ff40c1546f0 0x7ff40c072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff409400640 1 -- 172.21.15.81:0/4282533918 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3930787378 0 0) 0x7ff40c15db00 con 0x7ff40c0732f0 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff409400640 1 -- 172.21.15.81:0/4282533918 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3f0003530 con 0x7ff40c0732f0 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40b200640 1 -- 172.21.15.81:0/4282533918 >> v2:172.21.15.106:3300/0 conn(0x7ff40c1546f0 msgr2=0x7ff40c072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40b200640 1 --2- 172.21.15.81:0/4282533918 >> v2:172.21.15.106:3300/0 conn(0x7ff40c1546f0 0x7ff40c072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40b200640 1 -- 172.21.15.81:0/4282533918 >> v1:172.21.15.81:6789/0 conn(0x7ff40c0732f0 legacy=0x7ff40c0c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.755+0000 7ff40b200640 1 -- 172.21.15.81:0/4282533918 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c15ae60 con 0x7ff40c15a510 2024-05-27T06:56:46.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff40b200640 1 --2- 172.21.15.81:0/4282533918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c15a510 0x7ff40c15a920 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff40c06d950 tx=0x7ff40401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff564bf377d366ef server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff409400640 1 -- 172.21.15.81:0/4282533918 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4040b4070 con 0x7ff40c15a510 2024-05-27T06:56:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff409400640 1 -- 172.21.15.81:0/4282533918 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4040af600 con 0x7ff40c15a510 2024-05-27T06:56:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff409400640 1 -- 172.21.15.81:0/4282533918 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4040af9d0 con 0x7ff40c15a510 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 -- 172.21.15.81:0/4282533918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c15a510 msgr2=0x7ff40c15a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 --2- 172.21.15.81:0/4282533918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c15a510 0x7ff40c15a920 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff40c06d950 tx=0x7ff40401f110 comp rx=0 tx=0).stop 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 -- 172.21.15.81:0/4282533918 shutdown_connections 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 --2- 172.21.15.81:0/4282533918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c15a510 0x7ff40c15a920 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 --2- 172.21.15.81:0/4282533918 >> v2:172.21.15.106:3300/0 conn(0x7ff40c1546f0 0x7ff40c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 -- 172.21.15.81:0/4282533918 >> 172.21.15.81:0/4282533918 conn(0x7ff40c070700 msgr2=0x7ff40c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 -- 172.21.15.81:0/4282533918 shutdown_connections 2024-05-27T06:56:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.759+0000 7ff411400640 1 -- 172.21.15.81:0/4282533918 wait complete. 2024-05-27T06:56:46.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 Processor -- start 2024-05-27T06:56:46.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 -- start start 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 0x7ff40c14ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c15a510 0x7ff40c1480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c15e0a0 con 0x7ff40c1546f0 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c15ddc0 con 0x7ff40c0732f0 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c15e380 con 0x7ff40c15a510 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55022/0 (socket says 172.21.15.81:55022) 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 -- 172.21.15.81:0/1261578311 learned_addr learned my addr 172.21.15.81:0/1261578311 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff40a800640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 0x7ff40c14ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff40b200640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c15a510 0x7ff40c1480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 -- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c15a510 msgr2=0x7ff40c1480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c15a510 0x7ff40c1480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 -- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 msgr2=0x7ff40c14ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 0x7ff40c14ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 -- 172.21.15.81:0/1261578311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c1f2ff0 con 0x7ff40c1546f0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff40a800640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 0x7ff40c14ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff409e00640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800c460 tx=0x7ff3f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8017070 con 0x7ff40c1546f0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f8005cf0 con 0x7ff40c1546f0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f80072f0 con 0x7ff40c1546f0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.763+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c14f8e0 con 0x7ff40c1546f0 2024-05-27T06:56:46.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c1f3550 con 0x7ff40c1546f0 2024-05-27T06:56:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff40c14f630 con 0x7ff40c1546f0 2024-05-27T06:56:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff3f801a070 con 0x7ff40c1546f0 2024-05-27T06:56:46.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff402a00640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 0x7ff3ec071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:46.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff3f808e8f0 con 0x7ff40c1546f0 2024-05-27T06:56:46.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.767+0000 7ff40a800640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 0x7ff3ec071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:46.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.771+0000 7ff40a800640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 0x7ff3ec071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009b10 tx=0x7ff3fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:46.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.771+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff40c1f2ff0 con 0x7ff40c1546f0 2024-05-27T06:56:46.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff40c06a560 con 0x7ff40c1546f0 2024-05-27T06:56:46.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff402a00640 1 -- 172.21.15.81:0/1261578311 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff3f805b610 con 0x7ff40c1546f0 2024-05-27T06:56:46.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:46.954 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:46.954 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 msgr2=0x7ff3ec071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 0x7ff3ec071910 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009b10 tx=0x7ff3fc008040 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 msgr2=0x7ff40c0c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800c460 tx=0x7ff3f800c920 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 shutdown_connections 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff40c15a510 0x7ff40c1480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff3ec06f400 0x7ff3ec071910 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff40c1546f0 0x7ff40c0c8180 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 --2- 172.21.15.81:0/1261578311 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff40c0732f0 0x7ff40c14ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:46.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 >> 172.21.15.81:0/1261578311 conn(0x7ff40c070700 msgr2=0x7ff40c153800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:46.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 shutdown_connections 2024-05-27T06:56:46.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:46.963+0000 7ff411400640 1 -- 172.21.15.81:0/1261578311 wait complete. 2024-05-27T06:56:46.975 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:48.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:49.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 Processor -- start 2024-05-27T06:56:49.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 -- start start 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f324c0ca5e0 0x7f324c1605b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f324c1651c0 0x7f324c1675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f324c071f80 con 0x7f324c15c930 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f324c071a50 con 0x7f324c1651c0 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3252c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f324c15cd40 con 0x7f324c0ca5e0 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f324c1651c0 0x7f324c1675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f324c1651c0 0x7f324c1675e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55038/0 (socket says 172.21.15.81:55038) 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 -- 172.21.15.81:0/1789440947 learned_addr learned my addr 172.21.15.81:0/1789440947 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3243400640 1 --2- 172.21.15.81:0/1789440947 >> v2:172.21.15.106:3300/0 conn(0x7f324c0ca5e0 0x7f324c1605b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 -- 172.21.15.81:0/1789440947 >> v2:172.21.15.106:3300/0 conn(0x7f324c0ca5e0 msgr2=0x7f324c1605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:49.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 --2- 172.21.15.81:0/1789440947 >> v2:172.21.15.106:3300/0 conn(0x7f324c0ca5e0 0x7f324c1605b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 -- 172.21.15.81:0/1789440947 >> v1:172.21.15.106:6789/0 conn(0x7f324c15c930 legacy=0x7f324c15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.139+0000 7f3250c00640 1 -- 172.21.15.81:0/1789440947 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c071ca0 con 0x7f324c1651c0 2024-05-27T06:56:49.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3250c00640 1 --2- 172.21.15.81:0/1789440947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c1651c0 0x7f324c1675e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f324c070660 tx=0x7f324401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a278e01ed1a31829 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:49.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f324b400640 1 -- 172.21.15.81:0/1789440947 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32440a5790 con 0x7f324c1651c0 2024-05-27T06:56:49.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f324b400640 1 -- 172.21.15.81:0/1789440947 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32440a5930 con 0x7f324c1651c0 2024-05-27T06:56:49.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f324b400640 1 -- 172.21.15.81:0/1789440947 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32440a5d00 con 0x7f324c1651c0 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 -- 172.21.15.81:0/1789440947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c1651c0 msgr2=0x7f324c1675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 --2- 172.21.15.81:0/1789440947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c1651c0 0x7f324c1675e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f324c070660 tx=0x7f324401c8d0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 -- 172.21.15.81:0/1789440947 shutdown_connections 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 --2- 172.21.15.81:0/1789440947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c1651c0 0x7f324c1675e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 --2- 172.21.15.81:0/1789440947 >> v2:172.21.15.106:3300/0 conn(0x7f324c0ca5e0 0x7f324c1605b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 -- 172.21.15.81:0/1789440947 >> 172.21.15.81:0/1789440947 conn(0x7f324c070f30 msgr2=0x7f324c071340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 -- 172.21.15.81:0/1789440947 shutdown_connections 2024-05-27T06:56:49.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.143+0000 7f3252c00640 1 -- 172.21.15.81:0/1789440947 wait complete. 2024-05-27T06:56:49.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 Processor -- start 2024-05-27T06:56:49.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 -- start start 2024-05-27T06:56:49.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f324c1651c0 0x7f324c1ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f324c1f9910 0x7f324c148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f324c156fa0 con 0x7f324c15c930 2024-05-27T06:56:49.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f324c0c5970 con 0x7f324c1651c0 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3252c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f324c157120 con 0x7f324c1f9910 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55052/0 (socket says 172.21.15.81:55052) 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 -- 172.21.15.81:0/1708773639 learned_addr learned my addr 172.21.15.81:0/1708773639 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3243400640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f324c1651c0 0x7f324c1ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f3250c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f324c1f9910 0x7f324c148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 -- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f324c1f9910 msgr2=0x7f324c148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f324c1f9910 0x7f324c148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 -- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f324c1651c0 msgr2=0x7f324c1ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f324c1651c0 0x7f324c1ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 -- 172.21.15.81:0/1708773639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c1efbe0 con 0x7f324c15c930 2024-05-27T06:56:49.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.147+0000 7f324be00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c3c0 tx=0x7f323c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c017070 con 0x7f324c15c930 2024-05-27T06:56:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f323c0052d0 con 0x7f324c15c930 2024-05-27T06:56:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c003820 con 0x7f324c15c930 2024-05-27T06:56:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c14ace0 con 0x7f324c15c930 2024-05-27T06:56:49.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c14b280 con 0x7f324c15c930 2024-05-27T06:56:49.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f324c06a560 con 0x7f324c15c930 2024-05-27T06:56:49.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f323c01a070 con 0x7f324c15c930 2024-05-27T06:56:49.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 0x7f3228071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:49.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f323c08e7c0 con 0x7f324c15c930 2024-05-27T06:56:49.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.151+0000 7f3243400640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 0x7f3228071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:49.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.155+0000 7f3243400640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 0x7f3228071910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f324c1544f0 tx=0x7f3234008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:49.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.159+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f323c005480 con 0x7f324c15c930 2024-05-27T06:56:49.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.347+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f324c06d650 con 0x7f324c15c930 2024-05-27T06:56:49.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.347+0000 7f3248c00640 1 -- 172.21.15.81:0/1708773639 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f323c063410 con 0x7f324c15c930 2024-05-27T06:56:49.339 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:49.339 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:49.341 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:49.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 msgr2=0x7f3228071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 0x7f3228071910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f324c1544f0 tx=0x7f3234008040 comp rx=0 tx=0).stop 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 msgr2=0x7f324c0ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f323c00c3c0 tx=0x7f323c00c880 comp rx=0 tx=0).stop 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 shutdown_connections 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f324c1f9910 0x7f324c148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f322806f400 0x7f3228071910 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f324c1651c0 0x7f324c1ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 --2- 172.21.15.81:0/1708773639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f324c15c930 0x7f324c0ccd60 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:49.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 >> 172.21.15.81:0/1708773639 conn(0x7f324c070f30 msgr2=0x7f324c153610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:49.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 shutdown_connections 2024-05-27T06:56:49.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:49.351+0000 7f3252c00640 1 -- 172.21.15.81:0/1708773639 wait complete. 2024-05-27T06:56:49.360 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:51.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~47s 2024-05-27T06:56:51.361 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:51.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 Processor -- start 2024-05-27T06:56:51.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 -- start start 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4d340cfdb0 0x7f4d340d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340c78b0 con 0x7f4d340ca770 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340c6ee0 con 0x7f4d34154200 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d3b600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4d340c71c0 con 0x7f4d340cfdb0 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d38c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4d340ca770 0x7f4d340cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44534/0 (socket says 172.21.15.81:44534) 2024-05-27T06:56:51.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d38c00640 1 -- 172.21.15.81:0/2916544925 learned_addr learned my addr 172.21.15.81:0/2916544925 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:51.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d2be00640 1 --2- 172.21.15.81:0/2916544925 >> v2:172.21.15.106:3300/0 conn(0x7f4d340cfdb0 0x7f4d340d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039822183 0 0) 0x7f4d340c78b0 con 0x7f4d340ca770 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.531+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d18003530 con 0x7f4d340ca770 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3963654643 0 0) 0x7f4d18003530 con 0x7f4d340ca770 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 >> v2:172.21.15.106:3300/0 conn(0x7f4d340cfdb0 msgr2=0x7f4d340d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 --2- 172.21.15.81:0/2916544925 >> v2:172.21.15.106:3300/0 conn(0x7f4d340cfdb0 0x7f4d340d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 >> v1:172.21.15.106:6789/0 conn(0x7f4d34154200 legacy=0x7f4d34164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d340c7b70 con 0x7f4d340ca770 2024-05-27T06:56:51.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f4d24001700 con 0x7f4d340ca770 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 >> v1:172.21.15.81:6789/0 conn(0x7f4d340ca770 legacy=0x7f4d340cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 0x7f4d18003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18005260 0x7f4d1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340c71c0 con 0x7f4d18003a60 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1800c1f0 con 0x7f4d18004810 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1800c3f0 con 0x7f4d18005260 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 0x7f4d18003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d2be00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d39600640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18005260 0x7f4d1800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 -- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18005260 msgr2=0x7f4d1800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18005260 0x7f4d1800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 -- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d18004810 msgr2=0x7f4d18004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d18004810 0x7f4d18004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 -- 172.21.15.81:0/2916544925 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d18003530 con 0x7f4d18003a60 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d39600640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18005260 0x7f4d1800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d39600640 1 -- 172.21.15.81:0/2916544925 reap_dead start 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d38c00640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 0x7f4d18003ed0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400c240 tx=0x7f4d2401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc162090513c068 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24007d70 con 0x7f4d18003a60 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.535+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d240a9020 con 0x7f4d18003a60 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d33e00640 1 -- 172.21.15.81:0/2916544925 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24005b00 con 0x7f4d18003a60 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 msgr2=0x7f4d18003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 0x7f4d18003ed0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400c240 tx=0x7f4d2401cbc0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- 172.21.15.81:0/2916544925 shutdown_connections 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 --2- 172.21.15.81:0/2916544925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d18003a60 0x7f4d18003ed0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- 172.21.15.81:0/2916544925 >> 172.21.15.81:0/2916544925 conn(0x7f4d34068ae0 msgr2=0x7f4d34068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:51.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- 172.21.15.81:0/2916544925 shutdown_connections 2024-05-27T06:56:51.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- 172.21.15.81:0/2916544925 wait complete. 2024-05-27T06:56:51.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 Processor -- start 2024-05-27T06:56:51.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.539+0000 7f4d3b600640 1 -- start start 2024-05-27T06:56:51.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18003a60 0x7f4d34156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d341eb1d0 0x7f4d341f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340c7e50 con 0x7f4d3414d930 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4d34154a20 con 0x7f4d341eb1d0 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4d340c7fd0 con 0x7f4d18003a60 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55060/0 (socket says 172.21.15.81:55060) 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 -- 172.21.15.81:0/895144603 learned_addr learned my addr 172.21.15.81:0/895144603 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d38c00640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18003a60 0x7f4d34156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d2be00640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d341eb1d0 0x7f4d341f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 -- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18003a60 msgr2=0x7f4d34156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18003a60 0x7f4d34156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 -- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d341eb1d0 msgr2=0x7f4d341f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d341eb1d0 0x7f4d341f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 -- 172.21.15.81:0/895144603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d341f9f40 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d39600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d340724a0 tx=0x7f4d2c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c018070 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c003b80 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c00de80 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d34155b30 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d341e8790 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.543+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3406a560 con 0x7f4d3414d930 2024-05-27T06:56:51.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.547+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4d2c005070 con 0x7f4d3414d930 2024-05-27T06:56:51.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.547+0000 7f4d31600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 0x7f4d10071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:51.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.547+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4d2c08e900 con 0x7f4d3414d930 2024-05-27T06:56:51.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.547+0000 7f4d38c00640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 0x7f4d10071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:51.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.547+0000 7f4d38c00640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 0x7f4d10071940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d240aa000 tx=0x7f4d340c70b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.551+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4d2c05ae80 con 0x7f4d3414d930 2024-05-27T06:56:51.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.743+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d340cc5b0 con 0x7f4d3414d930 2024-05-27T06:56:51.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.743+0000 7f4d31600640 1 -- 172.21.15.81:0/895144603 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4d2c05f180 con 0x7f4d3414d930 2024-05-27T06:56:51.734 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:51.734 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:51.734 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 msgr2=0x7f4d10071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 0x7f4d10071940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d240aa000 tx=0x7f4d340c70b0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 msgr2=0x7f4d34159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:51.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4d340724a0 tx=0x7f4d2c00db40 comp rx=0 tx=0).stop 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 shutdown_connections 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4d1006f430 0x7f4d10071940 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d3414d930 0x7f4d34159220 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d341eb1d0 0x7f4d341f9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 --2- 172.21.15.81:0/895144603 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4d18003a60 0x7f4d34156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 >> 172.21.15.81:0/895144603 conn(0x7f4d34068ae0 msgr2=0x7f4d340ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 shutdown_connections 2024-05-27T06:56:51.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:51.747+0000 7f4d3b600640 1 -- 172.21.15.81:0/895144603 wait complete. 2024-05-27T06:56:51.756 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:53.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.923+0000 7fd315a00640 1 Processor -- start 2024-05-27T06:56:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.923+0000 7fd315a00640 1 -- start start 2024-05-27T06:56:53.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd315a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd3101546f0 0x7fd310072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:53.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd315a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd31015db00 con 0x7fd31015a510 2024-05-27T06:56:53.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd315a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd310073860 con 0x7fd3100732f0 2024-05-27T06:56:53.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd315a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd31015e6f0 con 0x7fd3101546f0 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd3101546f0 0x7fd310072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30f400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd31015a510 0x7fd31015a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44544/0 (socket says 172.21.15.81:44544) 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30f400640 1 -- 172.21.15.81:0/482740281 learned_addr learned my addr 172.21.15.81:0/482740281 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30e000640 1 -- 172.21.15.81:0/482740281 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605006940 0 0) 0x7fd31015db00 con 0x7fd31015a510 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30e000640 1 -- 172.21.15.81:0/482740281 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2f4003530 con 0x7fd31015a510 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30ea00640 1 -- 172.21.15.81:0/482740281 >> v1:172.21.15.106:6789/0 conn(0x7fd3100732f0 legacy=0x7fd3100c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30ea00640 1 -- 172.21.15.81:0/482740281 >> v1:172.21.15.81:6789/0 conn(0x7fd31015a510 legacy=0x7fd31015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:53.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.927+0000 7fd30ea00640 1 -- 172.21.15.81:0/482740281 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31015ddc0 con 0x7fd3101546f0 2024-05-27T06:56:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd30ea00640 1 --2- 172.21.15.81:0/482740281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3101546f0 0x7fd310072db0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097c0 tx=0x7fd30001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f34a7cf7732ff4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd30e000640 1 -- 172.21.15.81:0/482740281 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000b5070 con 0x7fd3101546f0 2024-05-27T06:56:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd30e000640 1 -- 172.21.15.81:0/482740281 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3000a8990 con 0x7fd3101546f0 2024-05-27T06:56:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd30e000640 1 -- 172.21.15.81:0/482740281 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3000a8d60 con 0x7fd3101546f0 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 -- 172.21.15.81:0/482740281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3101546f0 msgr2=0x7fd310072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 --2- 172.21.15.81:0/482740281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3101546f0 0x7fd310072db0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd3000097c0 tx=0x7fd30001ec90 comp rx=0 tx=0).stop 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 -- 172.21.15.81:0/482740281 shutdown_connections 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 --2- 172.21.15.81:0/482740281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3101546f0 0x7fd310072db0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 -- 172.21.15.81:0/482740281 >> 172.21.15.81:0/482740281 conn(0x7fd310070700 msgr2=0x7fd3100cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 -- 172.21.15.81:0/482740281 shutdown_connections 2024-05-27T06:56:53.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.931+0000 7fd315a00640 1 -- 172.21.15.81:0/482740281 wait complete. 2024-05-27T06:56:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 Processor -- start 2024-05-27T06:56:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- start start 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100732f0 0x7fd3100cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3101546f0 0x7fd310146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd31015e380 con 0x7fd31015a510 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd3100755b0 con 0x7fd3101546f0 2024-05-27T06:56:53.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd310154b00 con 0x7fd3100732f0 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55074/0 (socket says 172.21.15.81:55074) 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 -- 172.21.15.81:0/3086369854 learned_addr learned my addr 172.21.15.81:0/3086369854 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30ea00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100732f0 0x7fd3100cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd306000640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3101546f0 0x7fd310146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 -- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100732f0 msgr2=0x7fd3100cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100732f0 0x7fd3100cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 -- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3101546f0 msgr2=0x7fd310146bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3101546f0 0x7fd310146bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 -- 172.21.15.81:0/3086369854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd310147130 con 0x7fd31015a510 2024-05-27T06:56:53.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd30f400640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd30800cad0 tx=0x7fd30800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30800abd0 con 0x7fd31015a510 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd30800ad70 con 0x7fd31015a510 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3100ccda0 con 0x7fd31015a510 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308012600 con 0x7fd31015a510 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.935+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3101f3ec0 con 0x7fd31015a510 2024-05-27T06:56:53.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3100ccaf0 con 0x7fd31015a510 2024-05-27T06:56:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd3080127a0 con 0x7fd31015a510 2024-05-27T06:56:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd307400640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 0x7fd2ec071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd30808fb10 con 0x7fd31015a510 2024-05-27T06:56:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd30ea00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 0x7fd2ec071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.939+0000 7fd30ea00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 0x7fd2ec071940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd3000b2000 tx=0x7fd300008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:53.943+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd30805c770 con 0x7fd31015a510 2024-05-27T06:56:54.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.135+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd31006a560 con 0x7fd31015a510 2024-05-27T06:56:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.135+0000 7fd307400640 1 -- 172.21.15.81:0/3086369854 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd30805c070 con 0x7fd31015a510 2024-05-27T06:56:54.126 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:54.126 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:54.126 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 msgr2=0x7fd2ec071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 0x7fd2ec071940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd3000b2000 tx=0x7fd300008040 comp rx=0 tx=0).stop 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 msgr2=0x7fd3100c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd30800cad0 tx=0x7fd30800cf90 comp rx=0 tx=0).stop 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 shutdown_connections 2024-05-27T06:56:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd2ec06f430 0x7fd2ec071940 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd31015a510 0x7fd3100c60b0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3101546f0 0x7fd310146bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 --2- 172.21.15.81:0/3086369854 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd3100732f0 0x7fd3100cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 >> 172.21.15.81:0/3086369854 conn(0x7fd310070700 msgr2=0x7fd310150870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 shutdown_connections 2024-05-27T06:56:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:54.139+0000 7fd315a00640 1 -- 172.21.15.81:0/3086369854 wait complete. 2024-05-27T06:56:54.149 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:56:56.149 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 Processor -- start 2024-05-27T06:56:56.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 -- start start 2024-05-27T06:56:56.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdd980c6de0 0x7fdd980cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd980cc930 0x7fdd980ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd980755a0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fdd980ca9f0 con 0x7fdd980c6de0 2024-05-27T06:56:56.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd9da00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fdd980c7400 con 0x7fdd980cc930 2024-05-27T06:56:56.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd96000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdd9815a3e0 0x7fdd980ca2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60374/0 (socket says 172.21.15.81:60374) 2024-05-27T06:56:56.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd96000640 1 -- 172.21.15.81:0/120224354 learned_addr learned my addr 172.21.15.81:0/120224354 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:56.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.319+0000 7fdd96a00640 1 --2- 172.21.15.81:0/120224354 >> v2:172.21.15.81:3300/0 conn(0x7fdd980c6de0 0x7fdd980cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd97400640 1 --2- 172.21.15.81:0/120224354 >> v2:172.21.15.106:3300/0 conn(0x7fdd980cc930 0x7fdd980ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd95600640 1 -- 172.21.15.81:0/120224354 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331777077 0 0) 0x7fdd980755a0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd95600640 1 -- 172.21.15.81:0/120224354 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd7c003530 con 0x7fdd9815a3e0 2024-05-27T06:56:56.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd96a00640 1 -- 172.21.15.81:0/120224354 >> v2:172.21.15.106:3300/0 conn(0x7fdd980cc930 msgr2=0x7fdd980ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:56.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd96a00640 1 --2- 172.21.15.81:0/120224354 >> v2:172.21.15.106:3300/0 conn(0x7fdd980cc930 0x7fdd980ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd96a00640 1 -- 172.21.15.81:0/120224354 >> v1:172.21.15.81:6789/0 conn(0x7fdd9815a3e0 legacy=0x7fdd980ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd96a00640 1 -- 172.21.15.81:0/120224354 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd980c98c0 con 0x7fdd980c6de0 2024-05-27T06:56:56.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd96a00640 1 --2- 172.21.15.81:0/120224354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd980c6de0 0x7fdd980cbd40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdd9806fd80 tx=0x7fdd8801f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f699f7c012c0d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:56.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd95600640 1 -- 172.21.15.81:0/120224354 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880b4070 con 0x7fdd980c6de0 2024-05-27T06:56:56.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd95600640 1 -- 172.21.15.81:0/120224354 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd880af600 con 0x7fdd980c6de0 2024-05-27T06:56:56.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.323+0000 7fdd95600640 1 -- 172.21.15.81:0/120224354 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd880af9d0 con 0x7fdd980c6de0 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- 172.21.15.81:0/120224354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd980c6de0 msgr2=0x7fdd980cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 --2- 172.21.15.81:0/120224354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd980c6de0 0x7fdd980cbd40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdd9806fd80 tx=0x7fdd8801f110 comp rx=0 tx=0).stop 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- 172.21.15.81:0/120224354 shutdown_connections 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 --2- 172.21.15.81:0/120224354 >> v2:172.21.15.106:3300/0 conn(0x7fdd980cc930 0x7fdd980ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 --2- 172.21.15.81:0/120224354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd980c6de0 0x7fdd980cbd40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- 172.21.15.81:0/120224354 >> 172.21.15.81:0/120224354 conn(0x7fdd98068ab0 msgr2=0x7fdd98068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:56.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- 172.21.15.81:0/120224354 shutdown_connections 2024-05-27T06:56:56.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- 172.21.15.81:0/120224354 wait complete. 2024-05-27T06:56:56.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 Processor -- start 2024-05-27T06:56:56.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.327+0000 7fdd9da00640 1 -- start start 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd980c6de0 0x7fdd98151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd980cc930 0x7fdd981f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9815a7f0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980cb820 con 0x7fdd980cc930 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdd980c95c0 con 0x7fdd980c6de0 2024-05-27T06:56:56.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50446/0 (socket says 172.21.15.81:50446) 2024-05-27T06:56:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 -- 172.21.15.81:0/346572126 learned_addr learned my addr 172.21.15.81:0/346572126 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd96000640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd980cc930 0x7fdd981f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd96a00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd980c6de0 0x7fdd98151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 -- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd980c6de0 msgr2=0x7fdd98151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd980c6de0 0x7fdd98151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 -- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd980cc930 msgr2=0x7fdd981f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd980cc930 0x7fdd981f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 -- 172.21.15.81:0/346572126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd981e49d0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd97400640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdd90006e90 tx=0x7fdd9000d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd900125f0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd90019070 con 0x7fdd9815a3e0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd98151560 con 0x7fdd9815a3e0 2024-05-27T06:56:56.322 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd900129d0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd981e3c90 con 0x7fdd9815a3e0 2024-05-27T06:56:56.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.331+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd9806a560 con 0x7fdd9815a3e0 2024-05-27T06:56:56.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.335+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd90012b70 con 0x7fdd9815a3e0 2024-05-27T06:56:56.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.335+0000 7fdd8ea00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 0x7fdd78071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:56.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.335+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fdd9008f6e0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.335+0000 7fdd96a00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 0x7fdd78071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:56.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.335+0000 7fdd96a00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 0x7fdd78071940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd880b1000 tx=0x7fdd880aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:56.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.339+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdd9005bd80 con 0x7fdd9815a3e0 2024-05-27T06:56:56.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd981512f0 con 0x7fdd9815a3e0 2024-05-27T06:56:56.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd8ea00640 1 -- 172.21.15.81:0/346572126 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fdd90060080 con 0x7fdd9815a3e0 2024-05-27T06:56:56.522 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:56.522 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:56.522 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 msgr2=0x7fdd78071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 0x7fdd78071940 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdd880b1000 tx=0x7fdd880aa000 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 msgr2=0x7fdd981e4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdd90006e90 tx=0x7fdd9000d9e0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 shutdown_connections 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd7806f430 0x7fdd78071940 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd9815a3e0 0x7fdd981e4490 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd980cc930 0x7fdd981f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 --2- 172.21.15.81:0/346572126 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd980c6de0 0x7fdd98151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 >> 172.21.15.81:0/346572126 conn(0x7fdd98068ab0 msgr2=0x7fdd9814ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:56.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.531+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 shutdown_connections 2024-05-27T06:56:56.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:56.535+0000 7fdd9da00640 1 -- 172.21.15.81:0/346572126 wait complete. 2024-05-27T06:56:56.543 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:56:58.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~54s 2024-05-27T06:56:58.544 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:56:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 Processor -- start 2024-05-27T06:56:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 -- start start 2024-05-27T06:56:58.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f62c40736c0 0x7f62c4073c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f62c4074c00 0x7f62c4075010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62c415d710 con 0x7f62c406d120 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f62c406c8c0 con 0x7f62c40736c0 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62cbc00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f62c406cb60 con 0x7f62c4074c00 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f62c40736c0 0x7f62c4073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f62c406d120 0x7f62c415fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60376/0 (socket says 172.21.15.81:60376) 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9c00640 1 -- 172.21.15.81:0/324252894 learned_addr learned my addr 172.21.15.81:0/324252894 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62bbe00640 1 --2- 172.21.15.81:0/324252894 >> v2:172.21.15.106:3300/0 conn(0x7f62c4074c00 0x7f62c4075010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9200640 1 -- 172.21.15.81:0/324252894 >> v2:172.21.15.106:3300/0 conn(0x7f62c4074c00 msgr2=0x7f62c4075010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:56:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9200640 1 --2- 172.21.15.81:0/324252894 >> v2:172.21.15.106:3300/0 conn(0x7f62c4074c00 0x7f62c4075010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9200640 1 -- 172.21.15.81:0/324252894 >> v1:172.21.15.81:6789/0 conn(0x7f62c406d120 legacy=0x7f62c415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c9200640 1 -- 172.21.15.81:0/324252894 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c406ce40 con 0x7f62c40736c0 2024-05-27T06:56:58.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.715+0000 7f62c3e00640 1 -- 172.21.15.81:0/324252894 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283799591 0 0) 0x7f62c415d710 con 0x7f62c406d120 2024-05-27T06:56:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62c9200640 1 --2- 172.21.15.81:0/324252894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c40736c0 0x7f62c4073c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f62c4073650 tx=0x7f62b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e6499aae3da7f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:58.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62c3e00640 1 -- 172.21.15.81:0/324252894 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b40b0070 con 0x7f62c40736c0 2024-05-27T06:56:58.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62c3e00640 1 -- 172.21.15.81:0/324252894 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b40ab600 con 0x7f62c40736c0 2024-05-27T06:56:58.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62c3e00640 1 -- 172.21.15.81:0/324252894 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b40ab9d0 con 0x7f62c40736c0 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 -- 172.21.15.81:0/324252894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c40736c0 msgr2=0x7f62c4073c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 --2- 172.21.15.81:0/324252894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c40736c0 0x7f62c4073c40 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f62c4073650 tx=0x7f62b401c480 comp rx=0 tx=0).stop 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 -- 172.21.15.81:0/324252894 shutdown_connections 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 --2- 172.21.15.81:0/324252894 >> v2:172.21.15.106:3300/0 conn(0x7f62c4074c00 0x7f62c4075010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 --2- 172.21.15.81:0/324252894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c40736c0 0x7f62c4073c40 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 -- 172.21.15.81:0/324252894 >> 172.21.15.81:0/324252894 conn(0x7f62c40cefd0 msgr2=0x7f62c40cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 -- 172.21.15.81:0/324252894 shutdown_connections 2024-05-27T06:56:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.719+0000 7f62cbc00640 1 -- 172.21.15.81:0/324252894 wait complete. 2024-05-27T06:56:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 Processor -- start 2024-05-27T06:56:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 -- start start 2024-05-27T06:56:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c40736c0 0x7f62c4149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62c4074c00 0x7f62c40cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62c4156f50 con 0x7f62c406d120 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f62c4156dd0 con 0x7f62c40736c0 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62cbc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f62c416a060 con 0x7f62c4074c00 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50460/0 (socket says 172.21.15.81:50460) 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 -- 172.21.15.81:0/4053351207 learned_addr learned my addr 172.21.15.81:0/4053351207 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62bbe00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c40736c0 0x7f62c4149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9c00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62c4074c00 0x7f62c40cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 -- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62c4074c00 msgr2=0x7f62c40cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62c4074c00 0x7f62c40cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 -- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c40736c0 msgr2=0x7f62c4149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c40736c0 0x7f62c4149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.723+0000 7f62c9200640 1 -- 172.21.15.81:0/4053351207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c41f99c0 con 0x7f62c406d120 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c9200640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f62b401a0b0 tx=0x7f62b401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b40b0040 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b4004030 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c414ab00 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b40043e0 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c414d3b0 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c406a560 con 0x7f62c406d120 2024-05-27T06:56:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f62b4003620 con 0x7f62c406d120 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 0x7f62a0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f62b4129fd0 con 0x7f62c406d120 2024-05-27T06:56:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.727+0000 7f62bbe00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 0x7f62a0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:56:58.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.731+0000 7f62bbe00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 0x7f62a0071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62c4168020 tx=0x7f62ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:56:58.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.735+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f62b4016070 con 0x7f62c406d120 2024-05-27T06:56:58.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.923+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62c40719e0 con 0x7f62c406d120 2024-05-27T06:56:58.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.923+0000 7f62c1600640 1 -- 172.21.15.81:0/4053351207 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f62b40f65f0 con 0x7f62c406d120 2024-05-27T06:56:58.915 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:56:58.916 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:56:58.916 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:56:58.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 msgr2=0x7f62a0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 0x7f62a0071910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62c4168020 tx=0x7f62ac008040 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 msgr2=0x7f62c41f9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f62b401a0b0 tx=0x7f62b401acc0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 shutdown_connections 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f62c4074c00 0x7f62c40cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f62a006f400 0x7f62a0071910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62c40736c0 0x7f62c4149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 --2- 172.21.15.81:0/4053351207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62c406d120 0x7f62c41f9480 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 >> 172.21.15.81:0/4053351207 conn(0x7f62c40cefd0 msgr2=0x7f62c4156420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 shutdown_connections 2024-05-27T06:56:58.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:56:58.927+0000 7f62cbc00640 1 -- 172.21.15.81:0/4053351207 wait complete. 2024-05-27T06:56:58.938 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:00.938 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:01.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.107+0000 7ff24de00640 1 Processor -- start 2024-05-27T06:57:01.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.107+0000 7ff24de00640 1 -- start start 2024-05-27T06:57:01.100 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff248072430 0x7ff248159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff24815a040 0x7ff2480c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2480c60a0 0x7ff24815c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff2480c5040 con 0x7ff248072430 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff2480c75c0 con 0x7ff24815a040 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff24de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff2480c7dd0 con 0x7ff2480c60a0 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff248072430 0x7ff248159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff248072430 0x7ff248159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50472/0 (socket says 172.21.15.81:50472) 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 -- 172.21.15.81:0/3409502970 learned_addr learned my addr 172.21.15.81:0/3409502970 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff247400640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2480c60a0 0x7ff24815c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff23e000640 1 --2- 172.21.15.81:0/3409502970 >> v2:172.21.15.106:3300/0 conn(0x7ff24815a040 0x7ff2480c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 -- 172.21.15.81:0/3409502970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2480c60a0 msgr2=0x7ff24815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2480c60a0 0x7ff24815c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 -- 172.21.15.81:0/3409502970 >> v2:172.21.15.106:3300/0 conn(0x7ff24815a040 msgr2=0x7ff2480c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:01.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 --2- 172.21.15.81:0/3409502970 >> v2:172.21.15.106:3300/0 conn(0x7ff24815a040 0x7ff2480c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 -- 172.21.15.81:0/3409502970 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff248072840 con 0x7ff248072430 2024-05-27T06:57:01.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.111+0000 7ff246a00640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff248159b00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff24806f550 tx=0x7ff23801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a18a5ba88128a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff246000640 1 -- 172.21.15.81:0/3409502970 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380b0070 con 0x7ff248072430 2024-05-27T06:57:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff246000640 1 -- 172.21.15.81:0/3409502970 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2380ab600 con 0x7ff248072430 2024-05-27T06:57:01.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff246000640 1 -- 172.21.15.81:0/3409502970 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380ab9d0 con 0x7ff248072430 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- 172.21.15.81:0/3409502970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 msgr2=0x7ff248159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff248159b00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff24806f550 tx=0x7ff23801c260 comp rx=0 tx=0).stop 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- 172.21.15.81:0/3409502970 shutdown_connections 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff2480c60a0 0x7ff24815c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 --2- 172.21.15.81:0/3409502970 >> v2:172.21.15.106:3300/0 conn(0x7ff24815a040 0x7ff2480c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 --2- 172.21.15.81:0/3409502970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff248159b00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- 172.21.15.81:0/3409502970 >> 172.21.15.81:0/3409502970 conn(0x7ff248070700 msgr2=0x7ff2480cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- 172.21.15.81:0/3409502970 shutdown_connections 2024-05-27T06:57:01.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- 172.21.15.81:0/3409502970 wait complete. 2024-05-27T06:57:01.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 Processor -- start 2024-05-27T06:57:01.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.115+0000 7ff24de00640 1 -- start start 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2480c60a0 0x7ff2481417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff24815a040 0x7ff24814df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff2480ce6f0 con 0x7ff248072430 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff2481595e0 con 0x7ff2480c60a0 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff2480c5610 con 0x7ff24815a040 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50488/0 (socket says 172.21.15.81:50488) 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 -- 172.21.15.81:0/2248793292 learned_addr learned my addr 172.21.15.81:0/2248793292 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:01.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 -- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff24815a040 msgr2=0x7ff24814df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff24815a040 0x7ff24814df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 -- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2480c60a0 msgr2=0x7ff2481417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2480c60a0 0x7ff2481417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 -- 172.21.15.81:0/2248793292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24813ff10 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff246a00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ae040 tx=0x7ff238019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380b0040 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2380b9070 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2380b48a0 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2481f3e10 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2480cd5f0 con 0x7ff248072430 2024-05-27T06:57:01.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.119+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24806a560 con 0x7ff248072430 2024-05-27T06:57:01.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.123+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff2380b4a40 con 0x7ff248072430 2024-05-27T06:57:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.123+0000 7ff23fe00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 0x7ff224071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.123+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff2381293c0 con 0x7ff248072430 2024-05-27T06:57:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.123+0000 7ff23e000640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 0x7ff224071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:01.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.123+0000 7ff23e000640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 0x7ff224071940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff24806dc10 tx=0x7ff230008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:01.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.127+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff238016110 con 0x7ff248072430 2024-05-27T06:57:01.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.315+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2480c5940 con 0x7ff248072430 2024-05-27T06:57:01.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff23fe00640 1 -- 172.21.15.81:0/2248793292 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff2380f60b0 con 0x7ff248072430 2024-05-27T06:57:01.309 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:01.309 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:01.309 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 msgr2=0x7ff224071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 0x7ff224071940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff24806dc10 tx=0x7ff230008040 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 msgr2=0x7ff2481f48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ae040 tx=0x7ff238019000 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 shutdown_connections 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff24815a040 0x7ff24814df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff22406f430 0x7ff224071940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2480c60a0 0x7ff2481417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 --2- 172.21.15.81:0/2248793292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff248072430 0x7ff2481f48d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 >> 172.21.15.81:0/2248793292 conn(0x7ff248070700 msgr2=0x7ff2480c7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 shutdown_connections 2024-05-27T06:57:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:01.319+0000 7ff24de00640 1 -- 172.21.15.81:0/2248793292 wait complete. 2024-05-27T06:57:01.330 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:03.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:03.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 Processor -- start 2024-05-27T06:57:03.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 -- start start 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa7e4074540 0x7fa7e4161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e4074f80 con 0x7fa7e4162210 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e4075260 con 0x7fa7e4074130 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ec800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e4073650 con 0x7fa7e4074540 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ea800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa7e4162210 0x7fa7e4164630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60390/0 (socket says 172.21.15.81:60390) 2024-05-27T06:57:03.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.499+0000 7fa7ea800640 1 -- 172.21.15.81:0/3786470433 learned_addr learned my addr 172.21.15.81:0/3786470433 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> v2:172.21.15.106:3300/0 conn(0x7fa7e4074540 0x7fa7e4161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1035426160 0 0) 0x7fa7e4074f80 con 0x7fa7e4162210 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7cc003530 con 0x7fa7e4162210 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270477540 0 0) 0x7fa7e4075260 con 0x7fa7e4074130 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7e4074f80 con 0x7fa7e4074130 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4084942146 0 0) 0x7fa7cc003530 con 0x7fa7e4162210 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 >> v2:172.21.15.106:3300/0 conn(0x7fa7e4074540 msgr2=0x7fa7e4161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 --2- 172.21.15.81:0/3786470433 >> v2:172.21.15.106:3300/0 conn(0x7fa7e4074540 0x7fa7e4161cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 >> v1:172.21.15.106:6789/0 conn(0x7fa7e4074130 legacy=0x7fa7e406d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4073930 con 0x7fa7e4162210 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fa7e0001a80 con 0x7fa7e4162210 2024-05-27T06:57:03.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 >> v1:172.21.15.81:6789/0 conn(0x7fa7e4162210 legacy=0x7fa7e4164630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc003c40 0x7fa7cc004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 0x7fa7cc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7cc005290 0x7fa7cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4073650 con 0x7fa7cc004810 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa7cc00c240 con 0x7fa7cc005290 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa7cc00c440 con 0x7fa7cc003c40 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> v2:172.21.15.106:3300/0 conn(0x7fa7e4074540 0x7fa7e4161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc003c40 0x7fa7cc004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 0x7fa7cc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7ea800640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7cc005290 0x7fa7cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 -- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc003c40 msgr2=0x7fa7cc004090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc003c40 0x7fa7cc004090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 -- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7cc005290 msgr2=0x7fa7cc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7cc005290 0x7fa7cc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9400640 1 -- 172.21.15.81:0/3786470433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7cc003530 con 0x7fa7cc004810 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7ea800640 1 -- 172.21.15.81:0/3786470433 reap_dead start 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.503+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc003c40 0x7fa7cc004090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 0x7fa7cc004c20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa7e406ecd0 tx=0x7fa7d401cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af38dddebfff978 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:03.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4009860 con 0x7fa7cc004810 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d4004020 con 0x7fa7cc004810 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7e8a00640 1 -- 172.21.15.81:0/3786470433 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4004450 con 0x7fa7cc004810 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 msgr2=0x7fa7cc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 0x7fa7cc004c20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa7e406ecd0 tx=0x7fa7d401cb50 comp rx=0 tx=0).stop 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- 172.21.15.81:0/3786470433 shutdown_connections 2024-05-27T06:57:03.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3786470433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7cc004810 0x7fa7cc004c20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- 172.21.15.81:0/3786470433 >> 172.21.15.81:0/3786470433 conn(0x7fa7e4071770 msgr2=0x7fa7e40ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:03.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- 172.21.15.81:0/3786470433 shutdown_connections 2024-05-27T06:57:03.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- 172.21.15.81:0/3786470433 wait complete. 2024-05-27T06:57:03.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 Processor -- start 2024-05-27T06:57:03.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.507+0000 7fa7ec800640 1 -- start start 2024-05-27T06:57:03.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 0x7fa7e4156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e41f6340 0x7fa7e41575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e4075540 con 0x7fa7e414f450 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e40caff0 con 0x7fa7e41f6340 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e40c7570 con 0x7fa7cc004810 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50500/0 (socket says 172.21.15.81:50500) 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 -- 172.21.15.81:0/3734610097 learned_addr learned my addr 172.21.15.81:0/3734610097 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 0x7fa7e4156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7e9400640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e41f6340 0x7fa7e41575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 -- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 msgr2=0x7fa7e4156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 0x7fa7e4156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 -- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e41f6340 msgr2=0x7fa7e41575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e41f6340 0x7fa7e41575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 -- 172.21.15.81:0/3734610097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e4151dc0 con 0x7fa7e414f450 2024-05-27T06:57:03.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 0x7fa7e4156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:03.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ea800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40c8f40 tx=0x7fa7e0007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e000aae0 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e0018070 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e00133f0 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e40ca150 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e41588c0 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.511+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e4157d70 con 0x7fa7e414f450 2024-05-27T06:57:03.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.515+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa7e0013590 con 0x7fa7e414f450 2024-05-27T06:57:03.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.515+0000 7fa7de000640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 0x7fa7c8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:03.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.515+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa7e008fa50 con 0x7fa7e414f450 2024-05-27T06:57:03.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.515+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 0x7fa7c8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:03.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.515+0000 7fa7e9e00640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 0x7fa7c8071940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8005630 tx=0x7fa7d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:03.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.519+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa7e005c620 con 0x7fa7e414f450 2024-05-27T06:57:03.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.711+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7e406a560 con 0x7fa7e414f450 2024-05-27T06:57:03.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.711+0000 7fa7de000640 1 -- 172.21.15.81:0/3734610097 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fa7e005bf20 con 0x7fa7e414f450 2024-05-27T06:57:03.704 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:03.704 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:03.704 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 msgr2=0x7fa7c8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 0x7fa7c8071940 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8005630 tx=0x7fa7d8009040 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 msgr2=0x7fa7e40c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40c8f40 tx=0x7fa7e0007b10 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 shutdown_connections 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa7c806f430 0x7fa7c8071940 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7e414f450 0x7fa7e40c4530 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7e41f6340 0x7fa7e41575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 --2- 172.21.15.81:0/3734610097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7cc004810 0x7fa7e4156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:03.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 >> 172.21.15.81:0/3734610097 conn(0x7fa7e4071770 msgr2=0x7fa7e40c77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:03.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 shutdown_connections 2024-05-27T06:57:03.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:03.715+0000 7fa7ec800640 1 -- 172.21.15.81:0/3734610097 wait complete. 2024-05-27T06:57:03.725 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:04.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:57:04.657 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:57:04.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~60s 2024-05-27T06:57:05.726 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:05.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 Processor -- start 2024-05-27T06:57:05.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 -- start start 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2b280c5f70 0x7f2b28165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b28072840 con 0x7f2b28072430 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b280c5270 con 0x7f2b280c5b60 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b2d400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f2b280c5040 con 0x7f2b280c5f70 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b26800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2b28072430 0x7f2b28159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56598/0 (socket says 172.21.15.81:56598) 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b26800640 1 -- 172.21.15.81:0/983550308 learned_addr learned my addr 172.21.15.81:0/983550308 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:05.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b27200640 1 --2- 172.21.15.81:0/983550308 >> v2:172.21.15.81:3300/0 conn(0x7f2b280c5f70 0x7f2b28165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:05.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b25e00640 1 -- 172.21.15.81:0/983550308 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099029911 0 0) 0x7f2b28072840 con 0x7f2b28072430 2024-05-27T06:57:05.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b25e00640 1 -- 172.21.15.81:0/983550308 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b0c005740 con 0x7f2b28072430 2024-05-27T06:57:05.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b27200640 1 -- 172.21.15.81:0/983550308 >> v1:172.21.15.106:6789/0 conn(0x7f2b280c5b60 legacy=0x7f2b28161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:05.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b27200640 1 -- 172.21.15.81:0/983550308 >> v1:172.21.15.81:6789/0 conn(0x7f2b28072430 legacy=0x7f2b28159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:05.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.895+0000 7f2b27200640 1 -- 172.21.15.81:0/983550308 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b280c7810 con 0x7f2b280c5f70 2024-05-27T06:57:05.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b27200640 1 --2- 172.21.15.81:0/983550308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5f70 0x7f2b28165400 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009d90 tx=0x7f2b2001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bfd79176c61638 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:05.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b25e00640 1 -- 172.21.15.81:0/983550308 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b200b2070 con 0x7f2b280c5f70 2024-05-27T06:57:05.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b25e00640 1 -- 172.21.15.81:0/983550308 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b200a5730 con 0x7f2b280c5f70 2024-05-27T06:57:05.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b25e00640 1 -- 172.21.15.81:0/983550308 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b200a5b00 con 0x7f2b280c5f70 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b2d400640 1 -- 172.21.15.81:0/983550308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5f70 msgr2=0x7f2b28165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b2d400640 1 --2- 172.21.15.81:0/983550308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5f70 0x7f2b28165400 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009d90 tx=0x7f2b2001c970 comp rx=0 tx=0).stop 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b2d400640 1 -- 172.21.15.81:0/983550308 shutdown_connections 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b2d400640 1 --2- 172.21.15.81:0/983550308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5f70 0x7f2b28165400 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.899+0000 7f2b2d400640 1 -- 172.21.15.81:0/983550308 >> 172.21.15.81:0/983550308 conn(0x7f2b28070700 msgr2=0x7f2b280cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- 172.21.15.81:0/983550308 shutdown_connections 2024-05-27T06:57:05.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- 172.21.15.81:0/983550308 wait complete. 2024-05-27T06:57:05.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 Processor -- start 2024-05-27T06:57:05.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- start start 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b28072430 0x7f2b281f1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 0x7f2b28155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2b280c7dd0 con 0x7f2b280c5b60 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b280c75c0 con 0x7f2b280c5f70 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b2d400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2b280ce6f0 con 0x7f2b28072430 2024-05-27T06:57:05.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b1de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b1de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50052/0 (socket says 172.21.15.81:50052) 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.903+0000 7f2b1de00640 1 -- 172.21.15.81:0/1847165629 learned_addr learned my addr 172.21.15.81:0/1847165629 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b27200640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 0x7f2b28155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b26800640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b28072430 0x7f2b281f1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 -- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b28072430 msgr2=0x7f2b281f1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b28072430 0x7f2b281f1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 -- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 msgr2=0x7f2b28155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 0x7f2b28155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 -- 172.21.15.81:0/1847165629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b281f24d0 con 0x7f2b280c5b60 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b27200640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 0x7f2b28155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:05.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1de00640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c3f0 tx=0x7f2b1000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10017070 con 0x7f2b280c5b60 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b100052d0 con 0x7f2b280c5b60 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b281f51f0 con 0x7f2b280c5b60 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10003840 con 0x7f2b280c5b60 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b281f54b0 con 0x7f2b280c5b60 2024-05-27T06:57:05.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b2806a560 con 0x7f2b280c5b60 2024-05-27T06:57:05.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.907+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2b1001a070 con 0x7f2b280c5b60 2024-05-27T06:57:05.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.911+0000 7f2b1f400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 0x7f2b04071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.911+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f2b1008e8f0 con 0x7f2b280c5b60 2024-05-27T06:57:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.911+0000 7f2b26800640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 0x7f2b04071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:05.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.911+0000 7f2b26800640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 0x7f2b04071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2b18007a70 tx=0x7f2b18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:05.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:05.915+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2b10005480 con 0x7f2b280c5b60 2024-05-27T06:57:06.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.103+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b280c5940 con 0x7f2b280c5b60 2024-05-27T06:57:06.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.103+0000 7f2b1f400640 1 -- 172.21.15.81:0/1847165629 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f2b1005b550 con 0x7f2b280c5b60 2024-05-27T06:57:06.095 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:06.095 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:06.095 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 msgr2=0x7f2b04071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 0x7f2b04071910 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2b18007a70 tx=0x7f2b18009040 comp rx=0 tx=0).stop 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 msgr2=0x7f2b28155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c3f0 tx=0x7f2b1000c8b0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 shutdown_connections 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b0406f400 0x7f2b04071910 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b280c5f70 0x7f2b28155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b280c5b60 0x7f2b28155900 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 --2- 172.21.15.81:0/1847165629 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b28072430 0x7f2b281f1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:06.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 >> 172.21.15.81:0/1847165629 conn(0x7f2b28070700 msgr2=0x7f2b28159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:06.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 shutdown_connections 2024-05-27T06:57:06.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:06.107+0000 7f2b2d400640 1 -- 172.21.15.81:0/1847165629 wait complete. 2024-05-27T06:57:06.117 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:57:08.118 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:08.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 Processor -- start 2024-05-27T06:57:08.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 -- start start 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94d4072860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d41600e0 con 0x7f94d4156bd0 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d41603c0 con 0x7f94d4165030 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d9800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f94d4160c60 con 0x7f94d4072450 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d3e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f94d4156bd0 0x7f94d415ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56612/0 (socket says 172.21.15.81:56612) 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.287+0000 7f94d3e00640 1 -- 172.21.15.81:0/3658264242 learned_addr learned my addr 172.21.15.81:0/3658264242 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94d4072860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 615823462 0 0) 0x7f94d41600e0 con 0x7f94d4156bd0 2024-05-27T06:57:08.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94b8003530 con 0x7f94d4156bd0 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3799567946 0 0) 0x7f94b8003530 con 0x7f94d4156bd0 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 msgr2=0x7f94d4072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94d4072860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 >> v1:172.21.15.106:6789/0 conn(0x7f94d4165030 legacy=0x7f94d4167450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d41606a0 con 0x7f94d4156bd0 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f94c4001700 con 0x7f94d4156bd0 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 >> v1:172.21.15.81:6789/0 conn(0x7f94d4156bd0 legacy=0x7f94d415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8003a40 0x7f94b8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b8004810 0x7f94b8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94b80052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94d4160c60 con 0x7f94b8004810 2024-05-27T06:57:08.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f94b800bc20 con 0x7f94b8003a40 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f94b800be20 con 0x7f94d4072450 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b8004810 0x7f94b8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d3e00640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8003a40 0x7f94b8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94d8e00640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94b80052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 -- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 msgr2=0x7f94b80052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4072450 0x7f94b80052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 -- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8003a40 msgr2=0x7f94b8003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8003a40 0x7f94b8003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.291+0000 7f94cb400640 1 -- 172.21.15.81:0/3658264242 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b8003530 con 0x7f94b8004810 2024-05-27T06:57:08.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94cb400640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b8004810 0x7f94b8004c20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009ae0 tx=0x7f94bc01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6375bdac8019c7a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:08.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0b2070 con 0x7f94b8004810 2024-05-27T06:57:08.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc0a5680 con 0x7f94b8004810 2024-05-27T06:57:08.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d3400640 1 -- 172.21.15.81:0/3658264242 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0096b0 con 0x7f94b8004810 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- 172.21.15.81:0/3658264242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b8004810 msgr2=0x7f94b8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 --2- 172.21.15.81:0/3658264242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94b8004810 0x7f94b8004c20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009ae0 tx=0x7f94bc01ca90 comp rx=0 tx=0).stop 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d8e00640 1 -- 172.21.15.81:0/3658264242 reap_dead start 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- 172.21.15.81:0/3658264242 shutdown_connections 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- 172.21.15.81:0/3658264242 >> 172.21.15.81:0/3658264242 conn(0x7f94d4068880 msgr2=0x7f94d4068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- 172.21.15.81:0/3658264242 shutdown_connections 2024-05-27T06:57:08.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- 172.21.15.81:0/3658264242 wait complete. 2024-05-27T06:57:08.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 Processor -- start 2024-05-27T06:57:08.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.295+0000 7f94d9800640 1 -- start start 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8004810 0x7f94d41f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4157110 0x7f94d414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94d415d5e0 con 0x7f94d4159eb0 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f94d415ca40 con 0x7f94b8004810 2024-05-27T06:57:08.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f94d415d760 con 0x7f94d4157110 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50074/0 (socket says 172.21.15.81:50074) 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 -- 172.21.15.81:0/1969025984 learned_addr learned my addr 172.21.15.81:0/1969025984 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d3e00640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8004810 0x7f94d41f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d8e00640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4157110 0x7f94d414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 -- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4157110 msgr2=0x7f94d414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4157110 0x7f94d414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 -- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8004810 msgr2=0x7f94d41f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8004810 0x7f94d41f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 -- 172.21.15.81:0/1969025984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d41f35c0 con 0x7f94d4159eb0 2024-05-27T06:57:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94cb400640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0b0040 tx=0x7f94b8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0b2040 con 0x7f94d4159eb0 2024-05-27T06:57:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc00f000 con 0x7f94d4159eb0 2024-05-27T06:57:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d40c8690 con 0x7f94d4159eb0 2024-05-27T06:57:08.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc004240 con 0x7f94d4159eb0 2024-05-27T06:57:08.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d40c8c50 con 0x7f94d4159eb0 2024-05-27T06:57:08.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.299+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d41f2ca0 con 0x7f94d4159eb0 2024-05-27T06:57:08.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.303+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f94bc0c0020 con 0x7f94d4159eb0 2024-05-27T06:57:08.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.303+0000 7f94d0c00640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 0x7f94b0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:08.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.303+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f94bc12a360 con 0x7f94d4159eb0 2024-05-27T06:57:08.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.303+0000 7f94d3e00640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 0x7f94b0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:08.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.303+0000 7f94d3e00640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 0x7f94b0071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f94c4007fd0 tx=0x7f94c4001e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:08.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.307+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f94bc0167a0 con 0x7f94d4159eb0 2024-05-27T06:57:08.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94d40c3410 con 0x7f94d4159eb0 2024-05-27T06:57:08.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d0c00640 1 -- 172.21.15.81:0/1969025984 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f94bc0f69f0 con 0x7f94d4159eb0 2024-05-27T06:57:08.489 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:08.489 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:08.489 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 msgr2=0x7f94b0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 0x7f94b0071910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f94c4007fd0 tx=0x7f94c4001e80 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 msgr2=0x7f94d415c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0b0040 tx=0x7f94b8005840 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 shutdown_connections 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f94d4157110 0x7f94d414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f94b006f400 0x7f94b0071910 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d4159eb0 0x7f94d415c030 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 --2- 172.21.15.81:0/1969025984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94b8004810 0x7f94d41f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 >> 172.21.15.81:0/1969025984 conn(0x7f94d4068880 msgr2=0x7f94d4151810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 shutdown_connections 2024-05-27T06:57:08.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:08.499+0000 7f94d9800640 1 -- 172.21.15.81:0/1969025984 wait complete. 2024-05-27T06:57:08.510 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:10.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.682+0000 7f1187800640 1 Processor -- start 2024-05-27T06:57:10.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.682+0000 7f1187800640 1 -- start start 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1187800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f11800c6fd0 0x7f11800c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1187800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f118015db00 con 0x7f11801546f0 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1187800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1180073860 con 0x7f118015a510 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1187800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f118015e6f0 con 0x7f11800c6fd0 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1184e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f11801546f0 0x7f1180072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56626/0 (socket says 172.21.15.81:56626) 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f1184e00640 1 -- 172.21.15.81:0/3104307419 learned_addr learned my addr 172.21.15.81:0/3104307419 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117fe00640 1 --2- 172.21.15.81:0/3104307419 >> v2:172.21.15.81:3300/0 conn(0x7f11800c6fd0 0x7f11800c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:10.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117f400640 1 -- 172.21.15.81:0/3104307419 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096570178 0 0) 0x7f118015db00 con 0x7f11801546f0 2024-05-27T06:57:10.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117f400640 1 -- 172.21.15.81:0/3104307419 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1164003530 con 0x7f11801546f0 2024-05-27T06:57:10.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117fe00640 1 -- 172.21.15.81:0/3104307419 >> v1:172.21.15.106:6789/0 conn(0x7f118015a510 legacy=0x7f118015a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117fe00640 1 -- 172.21.15.81:0/3104307419 >> v1:172.21.15.81:6789/0 conn(0x7f11801546f0 legacy=0x7f1180072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.686+0000 7f117fe00640 1 -- 172.21.15.81:0/3104307419 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118015ddc0 con 0x7f11800c6fd0 2024-05-27T06:57:10.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f117fe00640 1 --2- 172.21.15.81:0/3104307419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800c6fd0 0x7f11800c73e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f116c0097c0 tx=0x7f116c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a863ad01afb785 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f117f400640 1 -- 172.21.15.81:0/3104307419 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c0b5070 con 0x7f11800c6fd0 2024-05-27T06:57:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f117f400640 1 -- 172.21.15.81:0/3104307419 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116c0a8990 con 0x7f11800c6fd0 2024-05-27T06:57:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f117f400640 1 -- 172.21.15.81:0/3104307419 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c0a8d60 con 0x7f11800c6fd0 2024-05-27T06:57:10.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 -- 172.21.15.81:0/3104307419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800c6fd0 msgr2=0x7f11800c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 --2- 172.21.15.81:0/3104307419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800c6fd0 0x7f11800c73e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f116c0097c0 tx=0x7f116c01ec90 comp rx=0 tx=0).stop 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 -- 172.21.15.81:0/3104307419 shutdown_connections 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 --2- 172.21.15.81:0/3104307419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11800c6fd0 0x7f11800c73e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 -- 172.21.15.81:0/3104307419 >> 172.21.15.81:0/3104307419 conn(0x7f1180070700 msgr2=0x7f11800cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 -- 172.21.15.81:0/3104307419 shutdown_connections 2024-05-27T06:57:10.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.690+0000 7f1187800640 1 -- 172.21.15.81:0/3104307419 wait complete. 2024-05-27T06:57:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 Processor -- start 2024-05-27T06:57:10.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- start start 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800c6fd0 0x7f1180145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 0x7f118015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f118015e380 con 0x7f118015a510 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f11800755b0 con 0x7f11800c6fd0 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1180154b00 con 0x7f11801546f0 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50092/0 (socket says 172.21.15.81:50092) 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 -- 172.21.15.81:0/4289605313 learned_addr learned my addr 172.21.15.81:0/4289605313 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:10.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f117fe00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 0x7f118015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:10.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1184e00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800c6fd0 0x7f1180145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 -- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 msgr2=0x7f118015d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 0x7f118015d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 -- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800c6fd0 msgr2=0x7f1180145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800c6fd0 0x7f1180145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 -- 172.21.15.81:0/4289605313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11801f0fc0 con 0x7f118015a510 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f117fe00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 0x7f118015d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1185800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f117800c370 tx=0x7f117800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178017070 con 0x7f118015a510 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11780052d0 con 0x7f118015a510 2024-05-27T06:57:10.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11801f1240 con 0x7f118015a510 2024-05-27T06:57:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1178003820 con 0x7f118015a510 2024-05-27T06:57:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.694+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11801f1800 con 0x7f118015a510 2024-05-27T06:57:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11801453b0 con 0x7f118015a510 2024-05-27T06:57:10.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f117801a070 con 0x7f118015a510 2024-05-27T06:57:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f117cc00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 0x7f1160071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f117808e900 con 0x7f118015a510 2024-05-27T06:57:10.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f1184e00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 0x7f1160071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:10.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.698+0000 7f1184e00640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 0x7f1160071940 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1170001770 tx=0x7f1170009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:10.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.702+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f11801f0fc0 con 0x7f118015a510 2024-05-27T06:57:10.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.894+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f118006a560 con 0x7f118015a510 2024-05-27T06:57:10.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.894+0000 7f117cc00640 1 -- 172.21.15.81:0/4289605313 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f1178005480 con 0x7f118015a510 2024-05-27T06:57:10.887 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:10.887 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:10.887 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:10.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 msgr2=0x7f1160071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 0x7f1160071940 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1170001770 tx=0x7f1170009040 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 msgr2=0x7f11800c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f117800c370 tx=0x7f117800c830 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 shutdown_connections 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f11801546f0 0x7f118015d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f116006f430 0x7f1160071940 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f118015a510 0x7f11800c7fb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 --2- 172.21.15.81:0/4289605313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11800c6fd0 0x7f1180145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 >> 172.21.15.81:0/4289605313 conn(0x7f1180070700 msgr2=0x7f118014a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 shutdown_connections 2024-05-27T06:57:10.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:10.898+0000 7f1187800640 1 -- 172.21.15.81:0/4289605313 wait complete. 2024-05-27T06:57:10.908 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:11.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~67s 2024-05-27T06:57:12.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 Processor -- start 2024-05-27T06:57:13.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 -- start start 2024-05-27T06:57:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb24c0ca770 0x7fb24c0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0cbd00 0x7fb24c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb24c0c78b0 con 0x7fb24c0d02f0 2024-05-27T06:57:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb24c0c6ee0 con 0x7fb24c0ca770 2024-05-27T06:57:13.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb252000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb24c0c71c0 con 0x7fb24c0cbd00 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb24b400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb24c0ca770 0x7fb24c0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb242a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0cbd00 0x7fb24c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb24be00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7fb24c0d02f0 0x7fb24c154200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:39552/0 (socket says 172.21.15.81:39552) 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.078+0000 7fb24be00640 1 -- 172.21.15.81:0/2660132442 learned_addr learned my addr 172.21.15.81:0/2660132442 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24b400640 1 -- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0cbd00 msgr2=0x7fb24c0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24b400640 1 --2- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0cbd00 0x7fb24c0cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24b400640 1 -- 172.21.15.81:0/2660132442 >> v1:172.21.15.106:6789/0 conn(0x7fb24c0d02f0 legacy=0x7fb24c154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24b400640 1 -- 172.21.15.81:0/2660132442 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c0c7b70 con 0x7fb24c0ca770 2024-05-27T06:57:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24b400640 1 --2- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0ca770 0x7fb24c0cb7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=690dac005cf9ada6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24aa00640 1 -- 172.21.15.81:0/2660132442 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0b2070 con 0x7fb24c0ca770 2024-05-27T06:57:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24aa00640 1 -- 172.21.15.81:0/2660132442 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c0a5940 con 0x7fb24c0ca770 2024-05-27T06:57:13.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.082+0000 7fb24aa00640 1 -- 172.21.15.81:0/2660132442 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0a5d10 con 0x7fb24c0ca770 2024-05-27T06:57:13.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0ca770 msgr2=0x7fb24c0cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 --2- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0ca770 0x7fb24c0cb7c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009d90 tx=0x7fb23c01c970 comp rx=0 tx=0).stop 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- 172.21.15.81:0/2660132442 shutdown_connections 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 --2- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0cbd00 0x7fb24c0cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 --2- 172.21.15.81:0/2660132442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0ca770 0x7fb24c0cb7c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- 172.21.15.81:0/2660132442 >> 172.21.15.81:0/2660132442 conn(0x7fb24c068ae0 msgr2=0x7fb24c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- 172.21.15.81:0/2660132442 shutdown_connections 2024-05-27T06:57:13.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- 172.21.15.81:0/2660132442 wait complete. 2024-05-27T06:57:13.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 Processor -- start 2024-05-27T06:57:13.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.086+0000 7fb252000640 1 -- start start 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 0x7fb24c06f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0cbd00 0x7fb24c159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c167590 con 0x7fb24c0d02f0 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c154b20 con 0x7fb24c0cbd00 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb24c167710 con 0x7fb24c0ca770 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 0x7fb24c06f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50094/0 (socket says 172.21.15.81:50094) 2024-05-27T06:57:13.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24be00640 1 -- 172.21.15.81:0/730994910 learned_addr learned my addr 172.21.15.81:0/730994910 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb242a00640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0cbd00 0x7fb24c159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 -- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0cbd00 msgr2=0x7fb24c159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0cbd00 0x7fb24c159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 -- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 msgr2=0x7fb24c1f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 -- 172.21.15.81:0/730994910 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c15a230 con 0x7fb24c0ca770 2024-05-27T06:57:13.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24be00640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb24b400640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 0x7fb24c06f000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0ac000 tx=0x7fb23c019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0b2050 con 0x7fb24c0ca770 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb23c0b3040 con 0x7fb24c0ca770 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0ad890 con 0x7fb24c0ca770 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c1f5e80 con 0x7fb24c0ca770 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c15d670 con 0x7fb24c0ca770 2024-05-27T06:57:13.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.090+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb24c15d400 con 0x7fb24c0ca770 2024-05-27T06:57:13.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.094+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb23c0c0020 con 0x7fb24c0ca770 2024-05-27T06:57:13.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.094+0000 7fb243e00640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 0x7fb228071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:13.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.094+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb23c12a2c0 con 0x7fb24c0ca770 2024-05-27T06:57:13.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.094+0000 7fb242a00640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 0x7fb228071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:13.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.094+0000 7fb242a00640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 0x7fb228071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb24c06fa40 tx=0x7fb234009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:13.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.098+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb23c0c7020 con 0x7fb24c0ca770 2024-05-27T06:57:13.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.290+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb24c06a560 con 0x7fb24c0ca770 2024-05-27T06:57:13.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.290+0000 7fb243e00640 1 -- 172.21.15.81:0/730994910 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fb24c1f5e80 con 0x7fb24c0ca770 2024-05-27T06:57:13.282 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:13.282 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:13.283 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 msgr2=0x7fb228071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 0x7fb228071910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb24c06fa40 tx=0x7fb234009190 comp rx=0 tx=0).stop 2024-05-27T06:57:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 msgr2=0x7fb24c06f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:13.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 0x7fb24c06f000 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0ac000 tx=0x7fb23c019710 comp rx=0 tx=0).stop 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 shutdown_connections 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb22806f400 0x7fb228071910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb24c0d02f0 0x7fb24c1f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb24c0cbd00 0x7fb24c159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 --2- 172.21.15.81:0/730994910 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb24c0ca770 0x7fb24c06f000 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 >> 172.21.15.81:0/730994910 conn(0x7fb24c068ae0 msgr2=0x7fb24c1638a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 shutdown_connections 2024-05-27T06:57:13.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:13.294+0000 7fb252000640 1 -- 172.21.15.81:0/730994910 wait complete. 2024-05-27T06:57:13.303 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:15.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 Processor -- start 2024-05-27T06:57:15.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 -- start start 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f61200ca2f0 0x7f61200caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f612015d300 0x7f6120160710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 0x7f61200d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f612015ce70 con 0x7f61200ca2f0 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f612015d710 con 0x7f61200cfdc0 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6127200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6120160c50 con 0x7f612015d300 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f61200ca2f0 0x7f61200caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f61200ca2f0 0x7f61200caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59946/0 (socket says 172.21.15.81:59946) 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 -- 172.21.15.81:0/1453460097 learned_addr learned my addr 172.21.15.81:0/1453460097 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:15.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6117400640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f612015d300 0x7f6120160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f6125200640 1 --2- 172.21.15.81:0/1453460097 >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 0x7f61200d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 -- 172.21.15.81:0/1453460097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f612015d300 msgr2=0x7f6120160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f612015d300 0x7f6120160710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 -- 172.21.15.81:0/1453460097 >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 msgr2=0x7f61200d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.474+0000 7f611fe00640 1 --2- 172.21.15.81:0/1453460097 >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 0x7f61200d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f611fe00640 1 -- 172.21.15.81:0/1453460097 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612006e3e0 con 0x7f61200ca2f0 2024-05-27T06:57:15.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f6125200640 1 --2- 172.21.15.81:0/1453460097 >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 0x7f61200d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f611fe00640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f61200caca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f61200d0980 tx=0x7f611001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74f2efc81f193b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f611f400640 1 -- 172.21.15.81:0/1453460097 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61100b2070 con 0x7f61200ca2f0 2024-05-27T06:57:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f611f400640 1 -- 172.21.15.81:0/1453460097 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61100a9070 con 0x7f61200ca2f0 2024-05-27T06:57:15.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.478+0000 7f611f400640 1 -- 172.21.15.81:0/1453460097 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61100ad880 con 0x7f61200ca2f0 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- 172.21.15.81:0/1453460097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 msgr2=0x7f61200caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f61200caca0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f61200d0980 tx=0x7f611001c490 comp rx=0 tx=0).stop 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- 172.21.15.81:0/1453460097 shutdown_connections 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 --2- 172.21.15.81:0/1453460097 >> v2:172.21.15.106:3300/0 conn(0x7f61200cfdc0 0x7f61200d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f612015d300 0x7f6120160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 --2- 172.21.15.81:0/1453460097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f61200caca0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- 172.21.15.81:0/1453460097 >> 172.21.15.81:0/1453460097 conn(0x7f6120068ae0 msgr2=0x7f6120068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- 172.21.15.81:0/1453460097 shutdown_connections 2024-05-27T06:57:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- 172.21.15.81:0/1453460097 wait complete. 2024-05-27T06:57:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 Processor -- start 2024-05-27T06:57:15.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.482+0000 7f6127200640 1 -- start start 2024-05-27T06:57:15.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61200cfdc0 0x7f61201f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612015d300 0x7f612015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f612006f700 con 0x7f61200ca2f0 2024-05-27T06:57:15.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f612015cb00 con 0x7f612015d300 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f612006f880 con 0x7f61200cfdc0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59960/0 (socket says 172.21.15.81:59960) 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 -- 172.21.15.81:0/3344124713 learned_addr learned my addr 172.21.15.81:0/3344124713 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6125200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612015d300 0x7f612015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6117400640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61200cfdc0 0x7f61201f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 -- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61200cfdc0 msgr2=0x7f61201f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61200cfdc0 0x7f61201f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 -- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612015d300 msgr2=0x7f612015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612015d300 0x7f612015ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 -- 172.21.15.81:0/3344124713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612015bd10 con 0x7f61200ca2f0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611fe00640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f611001cd30 tx=0x7f611001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:15.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61100bc020 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61100ade30 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61201f6a20 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61100b2040 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612015ba30 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.486+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612006a560 con 0x7f61200ca2f0 2024-05-27T06:57:15.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.490+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f61100bf020 con 0x7f61200ca2f0 2024-05-27T06:57:15.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.490+0000 7f611cc00640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 0x7f60fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:15.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.490+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6110129430 con 0x7f61200ca2f0 2024-05-27T06:57:15.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.490+0000 7f6117400640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 0x7f60fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:15.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.490+0000 7f6117400640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 0x7f60fc071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6120155b70 tx=0x7f6108008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:15.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.494+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f612015bd10 con 0x7f61200ca2f0 2024-05-27T06:57:15.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.686+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61200cefd0 con 0x7f61200ca2f0 2024-05-27T06:57:15.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.686+0000 7f611cc00640 1 -- 172.21.15.81:0/3344124713 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6110016110 con 0x7f61200ca2f0 2024-05-27T06:57:15.678 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:15.678 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:15.678 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:15.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 msgr2=0x7f60fc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 0x7f60fc071910 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6120155b70 tx=0x7f6108008040 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 msgr2=0x7f612015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f611001cd30 tx=0x7f611001acd0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 shutdown_connections 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f61200cfdc0 0x7f61201f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f60fc06f400 0x7f60fc071910 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612015d300 0x7f612015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 --2- 172.21.15.81:0/3344124713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61200ca2f0 0x7f612015a4e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 >> 172.21.15.81:0/3344124713 conn(0x7f6120068ae0 msgr2=0x7f6120155090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:15.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 shutdown_connections 2024-05-27T06:57:15.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:15.690+0000 7f6127200640 1 -- 172.21.15.81:0/3344124713 wait complete. 2024-05-27T06:57:15.701 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:17.703 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.870+0000 7fdd8ee00640 1 Processor -- start 2024-05-27T06:57:17.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.870+0000 7fdd8ee00640 1 -- start start 2024-05-27T06:57:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ee00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fdd880c5f70 0x7fdd88165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ee00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd88072840 con 0x7fdd88072430 2024-05-27T06:57:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ee00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd880c5270 con 0x7fdd880c5b60 2024-05-27T06:57:17.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ee00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fdd880c5040 con 0x7fdd880c5f70 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd87e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdd88072430 0x7fdd88159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43228/0 (socket says 172.21.15.81:43228) 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd87e00640 1 -- 172.21.15.81:0/2287652980 learned_addr learned my addr 172.21.15.81:0/2287652980 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ce00640 1 --2- 172.21.15.81:0/2287652980 >> v2:172.21.15.106:3300/0 conn(0x7fdd880c5f70 0x7fdd88165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd87400640 1 -- 172.21.15.81:0/2287652980 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2773801437 0 0) 0x7fdd88072840 con 0x7fdd88072430 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd87400640 1 -- 172.21.15.81:0/2287652980 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd6c003530 con 0x7fdd88072430 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ce00640 1 -- 172.21.15.81:0/2287652980 >> v1:172.21.15.106:6789/0 conn(0x7fdd880c5b60 legacy=0x7fdd88161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ce00640 1 -- 172.21.15.81:0/2287652980 >> v1:172.21.15.81:6789/0 conn(0x7fdd88072430 legacy=0x7fdd88159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:17.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.874+0000 7fdd8ce00640 1 -- 172.21.15.81:0/2287652980 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd880c7810 con 0x7fdd880c5f70 2024-05-27T06:57:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ce00640 1 --2- 172.21.15.81:0/2287652980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd88165400 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009d90 tx=0x7fdd8001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdc6c42f11dbd416 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd87400640 1 -- 172.21.15.81:0/2287652980 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd800b2070 con 0x7fdd880c5f70 2024-05-27T06:57:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd87400640 1 -- 172.21.15.81:0/2287652980 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd800a5830 con 0x7fdd880c5f70 2024-05-27T06:57:17.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd87400640 1 -- 172.21.15.81:0/2287652980 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd800a5be0 con 0x7fdd880c5f70 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- 172.21.15.81:0/2287652980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 msgr2=0x7fdd88165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/2287652980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd88165400 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdd80009d90 tx=0x7fdd8001c970 comp rx=0 tx=0).stop 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- 172.21.15.81:0/2287652980 shutdown_connections 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/2287652980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd88165400 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- 172.21.15.81:0/2287652980 >> 172.21.15.81:0/2287652980 conn(0x7fdd88070700 msgr2=0x7fdd880cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- 172.21.15.81:0/2287652980 shutdown_connections 2024-05-27T06:57:17.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- 172.21.15.81:0/2287652980 wait complete. 2024-05-27T06:57:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 Processor -- start 2024-05-27T06:57:17.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.878+0000 7fdd8ee00640 1 -- start start 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd880c5b60 0x7fdd8815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd880cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880c7dd0 con 0x7fdd88072430 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdd880c75c0 con 0x7fdd880c5f70 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fdd880ce6f0 con 0x7fdd880c5b60 2024-05-27T06:57:17.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59970/0 (socket says 172.21.15.81:59970) 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 -- 172.21.15.81:0/248491744 learned_addr learned my addr 172.21.15.81:0/248491744 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd7f400640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd880c5b60 0x7fdd8815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ce00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd880cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 -- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd880c5b60 msgr2=0x7fdd8815a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd880c5b60 0x7fdd8815a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 -- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 msgr2=0x7fdd880cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd880cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:17.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 -- 172.21.15.81:0/248491744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd88158230 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd87e00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c430 tx=0x7fdd7800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78017070 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd78005cf0 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd780072d0 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd88151470 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd88158420 con 0x7fdd88072430 2024-05-27T06:57:17.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.882+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd881585f0 con 0x7fdd88072430 2024-05-27T06:57:17.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.886+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fdd7801a070 con 0x7fdd88072430 2024-05-27T06:57:17.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.886+0000 7fdd84c00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 0x7fdd64071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:17.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.886+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fdd7808e840 con 0x7fdd88072430 2024-05-27T06:57:17.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.886+0000 7fdd7f400640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 0x7fdd64071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:17.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.886+0000 7fdd7f400640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 0x7fdd64071940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd880c91b0 tx=0x7fdd70009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:17.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:17.890+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fdd7805ae50 con 0x7fdd88072430 2024-05-27T06:57:18.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.082+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd8806a560 con 0x7fdd88072430 2024-05-27T06:57:18.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.082+0000 7fdd84c00640 1 -- 172.21.15.81:0/248491744 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fdd7805f150 con 0x7fdd88072430 2024-05-27T06:57:18.075 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:18.075 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:18.076 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:18.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 msgr2=0x7fdd64071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 0x7fdd64071940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd880c91b0 tx=0x7fdd70009040 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 msgr2=0x7fdd88151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800c430 tx=0x7fdd7800c8f0 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 shutdown_connections 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fdd880c5b60 0x7fdd8815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fdd6406f430 0x7fdd64071940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdd880c5f70 0x7fdd880cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 --2- 172.21.15.81:0/248491744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd88072430 0x7fdd88151c00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 >> 172.21.15.81:0/248491744 conn(0x7fdd88070700 msgr2=0x7fdd880ca190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 shutdown_connections 2024-05-27T06:57:18.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:18.086+0000 7fdd8ee00640 1 -- 172.21.15.81:0/248491744 wait complete. 2024-05-27T06:57:18.098 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:57:18.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~74s 2024-05-27T06:57:20.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.266+0000 7fc5bd400640 1 Processor -- start 2024-05-27T06:57:20.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.266+0000 7fc5bd400640 1 -- start start 2024-05-27T06:57:20.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5bd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc5b81546f0 0x7fc5b8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5bd400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b815db00 con 0x7fc5b815a510 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5bd400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b8073860 con 0x7fc5b80732f0 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5bd400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc5b815e6f0 con 0x7fc5b81546f0 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b7200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5b815a510 0x7fc5b815a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43234/0 (socket says 172.21.15.81:43234) 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b7200640 1 -- 172.21.15.81:0/1124777618 learned_addr learned my addr 172.21.15.81:0/1124777618 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:20.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b6800640 1 --2- 172.21.15.81:0/1124777618 >> v2:172.21.15.106:3300/0 conn(0x7fc5b81546f0 0x7fc5b8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1271732671 0 0) 0x7fc5b815db00 con 0x7fc5b815a510 2024-05-27T06:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc59c003530 con 0x7fc5b815a510 2024-05-27T06:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219317434 0 0) 0x7fc5b8073860 con 0x7fc5b80732f0 2024-05-27T06:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5b815db00 con 0x7fc5b80732f0 2024-05-27T06:57:20.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b6800640 1 -- 172.21.15.81:0/1124777618 >> v1:172.21.15.106:6789/0 conn(0x7fc5b80732f0 legacy=0x7fc5b80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b6800640 1 -- 172.21.15.81:0/1124777618 >> v1:172.21.15.81:6789/0 conn(0x7fc5b815a510 legacy=0x7fc5b815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.270+0000 7fc5b6800640 1 -- 172.21.15.81:0/1124777618 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b815ddc0 con 0x7fc5b81546f0 2024-05-27T06:57:20.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5b6800640 1 --2- 172.21.15.81:0/1124777618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b8072db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80097c0 tx=0x7fc5a801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9449e878af5581a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80b5070 con 0x7fc5b81546f0 2024-05-27T06:57:20.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a80a8990 con 0x7fc5b81546f0 2024-05-27T06:57:20.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5b5e00640 1 -- 172.21.15.81:0/1124777618 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80a8d60 con 0x7fc5b81546f0 2024-05-27T06:57:20.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 -- 172.21.15.81:0/1124777618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 msgr2=0x7fc5b8072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1124777618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b8072db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80097c0 tx=0x7fc5a801ec90 comp rx=0 tx=0).stop 2024-05-27T06:57:20.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 -- 172.21.15.81:0/1124777618 shutdown_connections 2024-05-27T06:57:20.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1124777618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b8072db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 -- 172.21.15.81:0/1124777618 >> 172.21.15.81:0/1124777618 conn(0x7fc5b8070700 msgr2=0x7fc5b80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:20.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 -- 172.21.15.81:0/1124777618 shutdown_connections 2024-05-27T06:57:20.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.274+0000 7fc5bd400640 1 -- 172.21.15.81:0/1124777618 wait complete. 2024-05-27T06:57:20.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 Processor -- start 2024-05-27T06:57:20.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- start start 2024-05-27T06:57:20.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5b80732f0 0x7fc5b81f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b81f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b815e380 con 0x7fc5b815a510 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b80755b0 con 0x7fc5b81546f0 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b8154b00 con 0x7fc5b80732f0 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59974/0 (socket says 172.21.15.81:59974) 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 -- 172.21.15.81:0/1339563404 learned_addr learned my addr 172.21.15.81:0/1339563404 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b6800640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5b80732f0 0x7fc5b81f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5ade00640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b81f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 -- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5b80732f0 msgr2=0x7fc5b81f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5b80732f0 0x7fc5b81f3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 -- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 msgr2=0x7fc5b81f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b81f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 -- 172.21.15.81:0/1339563404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b81f7750 con 0x7fc5b815a510 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5b7200640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000cab0 tx=0x7fc5b000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000ac30 con 0x7fc5b815a510 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b000add0 con 0x7fc5b815a510 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b814d1d0 con 0x7fc5b815a510 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0012600 con 0x7fc5b815a510 2024-05-27T06:57:20.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b81f3d20 con 0x7fc5b815a510 2024-05-27T06:57:20.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.278+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b806a560 con 0x7fc5b815a510 2024-05-27T06:57:20.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.282+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc5b00127a0 con 0x7fc5b815a510 2024-05-27T06:57:20.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.282+0000 7fc5af400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 0x7fc594071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:20.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.282+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fc5b008fb20 con 0x7fc5b815a510 2024-05-27T06:57:20.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.282+0000 7fc5b6800640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 0x7fc594071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:20.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.282+0000 7fc5b6800640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 0x7fc594071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80b2000 tx=0x7fc5a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:20.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.286+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc5b005c6f0 con 0x7fc5b815a510 2024-05-27T06:57:20.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5b81542c0 con 0x7fc5b815a510 2024-05-27T06:57:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5af400640 1 -- 172.21.15.81:0/1339563404 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fc5b005bff0 con 0x7fc5b815a510 2024-05-27T06:57:20.469 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:20.469 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:20.470 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:20.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 msgr2=0x7fc594071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 0x7fc594071910 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc5a80b2000 tx=0x7fc5a8008040 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 msgr2=0x7fc5b80c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000cab0 tx=0x7fc5b000cf70 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 shutdown_connections 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc59406f400 0x7fc594071910 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5b815a510 0x7fc5b80c9510 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b81546f0 0x7fc5b81f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 --2- 172.21.15.81:0/1339563404 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc5b80732f0 0x7fc5b81f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 >> 172.21.15.81:0/1339563404 conn(0x7fc5b8070700 msgr2=0x7fc5b8153850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 shutdown_connections 2024-05-27T06:57:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:20.478+0000 7fc5bd400640 1 -- 172.21.15.81:0/1339563404 wait complete. 2024-05-27T06:57:20.493 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:22.494 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:22.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 Processor -- start 2024-05-27T06:57:22.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 -- start start 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f70 0x7f762c165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c072840 con 0x7f762c0c5b60 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c0c5270 con 0x7f762c072430 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7631c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f762c0c5040 con 0x7f762c0c5f70 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7622000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f762c0c5b60 0x7f762c161cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43248/0 (socket says 172.21.15.81:43248) 2024-05-27T06:57:22.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f7622000640 1 -- 172.21.15.81:0/891489916 learned_addr learned my addr 172.21.15.81:0/891489916 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f762b400640 1 --2- 172.21.15.81:0/891489916 >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f70 0x7f762c165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.662+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 873258224 0 0) 0x7f762c072840 con 0x7f762c0c5b60 2024-05-27T06:57:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7610003530 con 0x7f762c0c5b60 2024-05-27T06:57:22.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 815165682 0 0) 0x7f762c0c5270 con 0x7f762c072430 2024-05-27T06:57:22.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762b400640 1 -- 172.21.15.81:0/891489916 >> v1:172.21.15.106:6789/0 conn(0x7f762c072430 legacy=0x7f762c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762b400640 1 -- 172.21.15.81:0/891489916 >> v1:172.21.15.81:6789/0 conn(0x7f762c0c5b60 legacy=0x7f762c161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762b400640 1 -- 172.21.15.81:0/891489916 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c0c7810 con 0x7f762c0c5f70 2024-05-27T06:57:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762b400640 1 --2- 172.21.15.81:0/891489916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c165400 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76240097c0 tx=0x7f762401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6f61d7a3463a65 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240b0070 con 0x7f762c0c5f70 2024-05-27T06:57:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76240ab600 con 0x7f762c0c5f70 2024-05-27T06:57:22.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.666+0000 7f762a000640 1 -- 172.21.15.81:0/891489916 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240ab9d0 con 0x7f762c0c5f70 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- 172.21.15.81:0/891489916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 msgr2=0x7f762c165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 --2- 172.21.15.81:0/891489916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c165400 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f76240097c0 tx=0x7f762401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- 172.21.15.81:0/891489916 shutdown_connections 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 --2- 172.21.15.81:0/891489916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c165400 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- 172.21.15.81:0/891489916 >> 172.21.15.81:0/891489916 conn(0x7f762c070700 msgr2=0x7f762c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- 172.21.15.81:0/891489916 shutdown_connections 2024-05-27T06:57:22.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- 172.21.15.81:0/891489916 wait complete. 2024-05-27T06:57:22.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 Processor -- start 2024-05-27T06:57:22.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 -- start start 2024-05-27T06:57:22.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f762c072430 0x7f762c148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:22.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.670+0000 7f7631c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f762c0c5b60 0x7f762c06c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:22.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:22.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f762c0c7dd0 con 0x7f762c0c5f70 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f762c0c75c0 con 0x7f762c072430 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f762c0ce6f0 con 0x7f762c0c5b60 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59986/0 (socket says 172.21.15.81:59986) 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 -- 172.21.15.81:0/1362210294 learned_addr learned my addr 172.21.15.81:0/1362210294 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7622000640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f762c0c5b60 0x7f762c06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762aa00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f762c072430 0x7f762c148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 -- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f762c0c5b60 msgr2=0x7f762c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f762c0c5b60 0x7f762c06c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 -- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f762c072430 msgr2=0x7f762c148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f762c072430 0x7f762c148e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 -- 172.21.15.81:0/1362210294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c1493c0 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f762b400640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f76240190b0 tx=0x7f762401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76240b0050 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76240bc070 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762c1f93f0 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624006e20 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762c1f99b0 con 0x7f762c0c5f70 2024-05-27T06:57:22.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f762c06a560 con 0x7f762c0c5f70 2024-05-27T06:57:22.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.674+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f76240b5810 con 0x7f762c0c5f70 2024-05-27T06:57:22.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.678+0000 7f7623400640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 0x7f7608071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:22.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.678+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f7624129af0 con 0x7f762c0c5f70 2024-05-27T06:57:22.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.678+0000 7f762aa00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 0x7f7608071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:22.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.678+0000 7f762aa00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 0x7f7608071910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f761c0012f0 tx=0x7f761c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:22.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.682+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f76240c6020 con 0x7f762c0c5f70 2024-05-27T06:57:22.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.870+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f762c1f9170 con 0x7f762c0c5f70 2024-05-27T06:57:22.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.870+0000 7f7623400640 1 -- 172.21.15.81:0/1362210294 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f762c1f9170 con 0x7f762c0c5f70 2024-05-27T06:57:22.864 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:22.864 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:22.864 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 msgr2=0x7f7608071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 0x7f7608071910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f761c0012f0 tx=0x7f761c009040 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 msgr2=0x7f762c152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f76240190b0 tx=0x7f762401ace0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 shutdown_connections 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f762c0c5b60 0x7f762c06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f760806f400 0x7f7608071910 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0c5f70 0x7f762c152580 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 --2- 172.21.15.81:0/1362210294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f762c072430 0x7f762c148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 >> 172.21.15.81:0/1362210294 conn(0x7f762c070700 msgr2=0x7f762c075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:22.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 shutdown_connections 2024-05-27T06:57:22.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:22.874+0000 7f7631c00640 1 -- 172.21.15.81:0/1362210294 wait complete. 2024-05-27T06:57:22.886 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:24.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:24.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~81s 2024-05-27T06:57:25.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.050+0000 7f3408c00640 1 Processor -- start 2024-05-27T06:57:25.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.050+0000 7f3408c00640 1 -- start start 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3408c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f340406c790 0x7f340406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3408c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f340415cbe0 con 0x7f340415c730 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3408c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f340415cec0 con 0x7f34040c8a30 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3408c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f34040c82c0 con 0x7f340406c790 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f33fbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f340406c790 0x7f340406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3401e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f340415c730 0x7f34040c9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43252/0 (socket says 172.21.15.81:43252) 2024-05-27T06:57:25.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3401e00640 1 -- 172.21.15.81:0/1181547054 learned_addr learned my addr 172.21.15.81:0/1181547054 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f33fbe00640 1 -- 172.21.15.81:0/1181547054 >> v1:172.21.15.106:6789/0 conn(0x7f34040c8a30 legacy=0x7f34041670b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f33fbe00640 1 -- 172.21.15.81:0/1181547054 >> v1:172.21.15.81:6789/0 conn(0x7f340415c730 legacy=0x7f34040c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f33fbe00640 1 -- 172.21.15.81:0/1181547054 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340415d1a0 con 0x7f340406c790 2024-05-27T06:57:25.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.054+0000 7f3401400640 1 -- 172.21.15.81:0/1181547054 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470006178 0 0) 0x7f340415cbe0 con 0x7f340415c730 2024-05-27T06:57:25.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f33fbe00640 1 --2- 172.21.15.81:0/1181547054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f340406cba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f34040c8440 tx=0x7f33ec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd4eb5587fb264b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3401400640 1 -- 172.21.15.81:0/1181547054 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0b0070 con 0x7f340406c790 2024-05-27T06:57:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3401400640 1 -- 172.21.15.81:0/1181547054 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ec0ab600 con 0x7f340406c790 2024-05-27T06:57:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3401400640 1 -- 172.21.15.81:0/1181547054 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0ab9d0 con 0x7f340406c790 2024-05-27T06:57:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 -- 172.21.15.81:0/1181547054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 msgr2=0x7f340406cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 --2- 172.21.15.81:0/1181547054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f340406cba0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f34040c8440 tx=0x7f33ec01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 -- 172.21.15.81:0/1181547054 shutdown_connections 2024-05-27T06:57:25.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 --2- 172.21.15.81:0/1181547054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f340406cba0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 -- 172.21.15.81:0/1181547054 >> 172.21.15.81:0/1181547054 conn(0x7f3404070700 msgr2=0x7f34040cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:25.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 -- 172.21.15.81:0/1181547054 shutdown_connections 2024-05-27T06:57:25.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.058+0000 7f3408c00640 1 -- 172.21.15.81:0/1181547054 wait complete. 2024-05-27T06:57:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 Processor -- start 2024-05-27T06:57:25.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- start start 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34040c8a30 0x7f3404143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 0x7f3404144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34040c9250 con 0x7f340406c790 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f34040c9c30 con 0x7f340415c730 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3404075440 con 0x7f34040c8a30 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60006/0 (socket says 172.21.15.81:60006) 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 -- 172.21.15.81:0/197960401 learned_addr learned my addr 172.21.15.81:0/197960401 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f33fbe00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34040c8a30 0x7f3404143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3402800640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 0x7f3404144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 -- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34040c8a30 msgr2=0x7f3404143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34040c8a30 0x7f3404143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 -- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 msgr2=0x7f3404144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 0x7f3404144370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 -- 172.21.15.81:0/197960401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f340414cbc0 con 0x7f340406c790 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3402800640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 0x7f3404144370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3401e00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c460 tx=0x7f33f400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4018070 con 0x7f340406c790 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f4007500 con 0x7f340406c790 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f340414c4f0 con 0x7f340406c790 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4004a30 con 0x7f340406c790 2024-05-27T06:57:25.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34040cc8f0 con 0x7f340406c790 2024-05-27T06:57:25.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.062+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f340406a560 con 0x7f340406c790 2024-05-27T06:57:25.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.066+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f33f401b070 con 0x7f340406c790 2024-05-27T06:57:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.066+0000 7f33fa000640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 0x7f33e0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.066+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f33f408e840 con 0x7f340406c790 2024-05-27T06:57:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.066+0000 7f33fbe00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 0x7f33e0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:25.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.066+0000 7f33fbe00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 0x7f33e0071940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0ad000 tx=0x7f33ec0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:25.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.070+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f33f405b410 con 0x7f340406c790 2024-05-27T06:57:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34040c5f30 con 0x7f340406c790 2024-05-27T06:57:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f33fa000640 1 -- 172.21.15.81:0/197960401 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f33f405ad10 con 0x7f340406c790 2024-05-27T06:57:25.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:25.253 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 msgr2=0x7f33e0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 0x7f33e0071940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0ad000 tx=0x7f33ec0a6000 comp rx=0 tx=0).stop 2024-05-27T06:57:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 msgr2=0x7f34041554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c460 tx=0x7f33f400c920 comp rx=0 tx=0).stop 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 shutdown_connections 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f34040c8a30 0x7f3404143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f33e006f430 0x7f33e0071940 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f340415c730 0x7f3404144370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 --2- 172.21.15.81:0/197960401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f340406c790 0x7f34041554e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 >> 172.21.15.81:0/197960401 conn(0x7f3404070700 msgr2=0x7f340415ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 shutdown_connections 2024-05-27T06:57:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:25.262+0000 7f3408c00640 1 -- 172.21.15.81:0/197960401 wait complete. 2024-05-27T06:57:25.275 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:27.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:27.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 Processor -- start 2024-05-27T06:57:27.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 -- start start 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd881684b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8815c710 con 0x7efd880ca2f0 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8806e560 con 0x7efd880caca0 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8fc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd8806eb20 con 0x7efd88166090 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8d200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efd880ca2f0 0x7efd881606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:48132/0 (socket says 172.21.15.81:48132) 2024-05-27T06:57:27.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8d200640 1 -- 172.21.15.81:0/3649142789 learned_addr learned my addr 172.21.15.81:0/3649142789 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8dc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd881684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064296100 0 0) 0x7efd8815c710 con 0x7efd880ca2f0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd70003530 con 0x7efd880ca2f0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910524082 0 0) 0x7efd8806e560 con 0x7efd880caca0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd8815c710 con 0x7efd880caca0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3133846006 0 0) 0x7efd70003530 con 0x7efd880ca2f0 2024-05-27T06:57:27.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 msgr2=0x7efd881684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd881684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 >> v1:172.21.15.106:6789/0 conn(0x7efd880caca0 legacy=0x7efd880cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8806e840 con 0x7efd880ca2f0 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7efd7c001700 con 0x7efd880ca2f0 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 >> v1:172.21.15.81:6789/0 conn(0x7efd880ca2f0 legacy=0x7efd881606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd70003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd700041d0 0x7efd70004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd70005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd7000bc00 con 0x7efd70004da0 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd7000bdb0 con 0x7efd700041d0 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd7000bfb0 con 0x7efd88166090 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8dc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd70005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd8d200640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd70003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.446+0000 7efd87e00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd700041d0 0x7efd70004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 -- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 msgr2=0x7efd70003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd70003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 -- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd700041d0 msgr2=0x7efd70004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd700041d0 0x7efd70004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 -- 172.21.15.81:0/3649142789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd70003530 con 0x7efd70004da0 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8d200640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd88166090 0x7efd70003c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:27.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd87e00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd700041d0 0x7efd70004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd70005230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd80009fc0 tx=0x7efd8001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451be3a6bf8f4f1d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd800b2070 con 0x7efd70004da0 2024-05-27T06:57:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd800a5ba0 con 0x7efd70004da0 2024-05-27T06:57:27.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd87400640 1 -- 172.21.15.81:0/3649142789 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd80009c10 con 0x7efd70004da0 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 -- 172.21.15.81:0/3649142789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 msgr2=0x7efd70005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 --2- 172.21.15.81:0/3649142789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd70005230 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd80009fc0 tx=0x7efd8001cca0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8dc00640 1 -- 172.21.15.81:0/3649142789 reap_dead start 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 -- 172.21.15.81:0/3649142789 shutdown_connections 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 -- 172.21.15.81:0/3649142789 >> 172.21.15.81:0/3649142789 conn(0x7efd88068ae0 msgr2=0x7efd88068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 -- 172.21.15.81:0/3649142789 shutdown_connections 2024-05-27T06:57:27.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.450+0000 7efd8fc00640 1 -- 172.21.15.81:0/3649142789 wait complete. 2024-05-27T06:57:27.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 Processor -- start 2024-05-27T06:57:27.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 -- start start 2024-05-27T06:57:27.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 0x7efd881561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 0x7efd8814b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd8815ccb0 con 0x7efd70004da0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd8806f910 con 0x7efd88159c30 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8fc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7efd8815cf90 con 0x7efd881f5cf0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36532/0 (socket says 172.21.15.81:36532) 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8d200640 1 -- 172.21.15.81:0/1183717022 learned_addr learned my addr 172.21.15.81:0/1183717022 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd8dc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 0x7efd8814b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.454+0000 7efd87e00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 0x7efd881561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 -- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 msgr2=0x7efd8814b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 0x7efd8814b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 -- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 msgr2=0x7efd881561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 0x7efd881561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 -- 172.21.15.81:0/1183717022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd88157b50 con 0x7efd70004da0 2024-05-27T06:57:27.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8dc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 0x7efd8814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd87e00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 0x7efd881561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8d200640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd7c011830 tx=0x7efd7c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c010040 con 0x7efd70004da0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c01a070 con 0x7efd70004da0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd881f2890 con 0x7efd70004da0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c00fd00 con 0x7efd70004da0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd881f2b50 con 0x7efd70004da0 2024-05-27T06:57:27.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8806a560 con 0x7efd70004da0 2024-05-27T06:57:27.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7efd7c01f460 con 0x7efd70004da0 2024-05-27T06:57:27.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.458+0000 7efd84c00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 0x7efd68071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:27.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.462+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7efd7c0927f0 con 0x7efd70004da0 2024-05-27T06:57:27.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.462+0000 7efd87e00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 0x7efd68071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:27.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.462+0000 7efd87e00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 0x7efd68071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c10 tx=0x7efd74009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:27.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.466+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7efd7c05f450 con 0x7efd70004da0 2024-05-27T06:57:27.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.658+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd8806ddf0 con 0x7efd70004da0 2024-05-27T06:57:27.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.658+0000 7efd84c00640 1 -- 172.21.15.81:0/1183717022 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7efd7c090e20 con 0x7efd70004da0 2024-05-27T06:57:27.650 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:27.650 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:27.650 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:27.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 msgr2=0x7efd68071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 0x7efd68071910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c10 tx=0x7efd74009040 comp rx=0 tx=0).stop 2024-05-27T06:57:27.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 msgr2=0x7efd88157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:27.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7efd7c011830 tx=0x7efd7c011cf0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 shutdown_connections 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7efd881f5cf0 0x7efd8814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7efd6806f400 0x7efd68071910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd88159c30 0x7efd881561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 --2- 172.21.15.81:0/1183717022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd70004da0 0x7efd88157610 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 >> 172.21.15.81:0/1183717022 conn(0x7efd88068ae0 msgr2=0x7efd880cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 shutdown_connections 2024-05-27T06:57:27.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:27.662+0000 7efd8fc00640 1 -- 172.21.15.81:0/1183717022 wait complete. 2024-05-27T06:57:27.673 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:29.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 Processor -- start 2024-05-27T06:57:29.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 -- start start 2024-05-27T06:57:29.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 0x7f6aa415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 0x7f6aa40ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa40755a0 con 0x7f6aa40c6de0 2024-05-27T06:57:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa40ca9f0 con 0x7f6aa40cc930 2024-05-27T06:57:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa8c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa40cc150 con 0x7f6aa40cbd40 2024-05-27T06:57:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa2800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 0x7f6aa40ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6a9be00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 0x7f6aa415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa1e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f6aa40c6de0 0x7f6aa40ca2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:41998/0 (socket says 172.21.15.81:41998) 2024-05-27T06:57:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa1e00640 1 -- 172.21.15.81:0/80313055 learned_addr learned my addr 172.21.15.81:0/80313055 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:29.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.842+0000 7f6aa2800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 0x7f6aa40ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:34180/0 (socket says 172.21.15.81:34180) 2024-05-27T06:57:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6a9be00640 1 -- 172.21.15.81:0/80313055 >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 msgr2=0x7f6aa40ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6aa1400640 1 -- 172.21.15.81:0/80313055 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606589957 0 0) 0x7f6aa40755a0 con 0x7f6aa40c6de0 2024-05-27T06:57:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6a9be00640 1 --2- 172.21.15.81:0/80313055 >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 0x7f6aa40ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6a9be00640 1 -- 172.21.15.81:0/80313055 >> v1:172.21.15.106:6789/0 conn(0x7f6aa40c6de0 legacy=0x7f6aa40ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:29.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6a9be00640 1 -- 172.21.15.81:0/80313055 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa40c7400 con 0x7f6aa40cbd40 2024-05-27T06:57:29.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6a9be00640 1 --2- 172.21.15.81:0/80313055 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 0x7f6aa415a3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0097c0 tx=0x7f6a8c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a335ca91b7854 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:29.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6aa1400640 1 -- 172.21.15.81:0/80313055 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0b5070 con 0x7f6aa40cbd40 2024-05-27T06:57:29.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6aa1400640 1 -- 172.21.15.81:0/80313055 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c0a8990 con 0x7f6aa40cbd40 2024-05-27T06:57:29.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.846+0000 7f6aa1400640 1 -- 172.21.15.81:0/80313055 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0a8d60 con 0x7f6aa40cbd40 2024-05-27T06:57:29.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- 172.21.15.81:0/80313055 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 msgr2=0x7f6aa415a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:29.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/80313055 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 0x7f6aa415a3e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c0097c0 tx=0x7f6a8c01ec90 comp rx=0 tx=0).stop 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- 172.21.15.81:0/80313055 shutdown_connections 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/80313055 >> v2:172.21.15.106:3300/0 conn(0x7f6aa40cc930 0x7f6aa40ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/80313055 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40cbd40 0x7f6aa415a3e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- 172.21.15.81:0/80313055 >> 172.21.15.81:0/80313055 conn(0x7f6aa4068ab0 msgr2=0x7f6aa4068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- 172.21.15.81:0/80313055 shutdown_connections 2024-05-27T06:57:29.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- 172.21.15.81:0/80313055 wait complete. 2024-05-27T06:57:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 Processor -- start 2024-05-27T06:57:29.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- start start 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40c6de0 0x7f6aa40ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa40cbd40 0x7f6aa4147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40cb820 con 0x7f6aa40cc930 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40c9ba0 con 0x7f6aa40cbd40 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.850+0000 7f6aa8c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa40c95c0 con 0x7f6aa40c6de0 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36538/0 (socket says 172.21.15.81:36538) 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 -- 172.21.15.81:0/4245302958 learned_addr learned my addr 172.21.15.81:0/4245302958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:29.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6a9be00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa40cbd40 0x7f6aa4147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa1e00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40c6de0 0x7f6aa40ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 -- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40c6de0 msgr2=0x7f6aa40ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40c6de0 0x7f6aa40ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 -- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa40cbd40 msgr2=0x7f6aa4147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa40cbd40 0x7f6aa4147630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 -- 172.21.15.81:0/4245302958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa40cec30 con 0x7f6aa40cc930 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa2800640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00ec10 tx=0x7f6a9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c018020 con 0x7f6aa40cc930 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9c00cce0 con 0x7f6aa40cc930 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa406dbb0 con 0x7f6aa40cc930 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0146c0 con 0x7f6aa40cc930 2024-05-27T06:57:29.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa406e170 con 0x7f6aa40cc930 2024-05-27T06:57:29.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa40ce470 con 0x7f6aa40cc930 2024-05-27T06:57:29.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.854+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6a9c01b020 con 0x7f6aa40cc930 2024-05-27T06:57:29.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.858+0000 7f6a9a000640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 0x7f6a80071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:29.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.858+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6a9c08ffe0 con 0x7f6aa40cc930 2024-05-27T06:57:29.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.858+0000 7f6aa1e00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 0x7f6a80071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:29.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.858+0000 7f6aa1e00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 0x7f6a80071940 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6a94000f70 tx=0x7f6a94007df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:29.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:29.862+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6a9c022020 con 0x7f6aa40cc930 2024-05-27T06:57:30.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.050+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6aa406d940 con 0x7f6aa40cc930 2024-05-27T06:57:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6a9a000640 1 -- 172.21.15.81:0/4245302958 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6a9c010040 con 0x7f6aa40cc930 2024-05-27T06:57:30.044 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:30.044 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:30.044 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:30.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 msgr2=0x7f6a80071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:30.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 0x7f6a80071940 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6a94000f70 tx=0x7f6a94007df0 comp rx=0 tx=0).stop 2024-05-27T06:57:30.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 msgr2=0x7f6aa40c53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:30.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00ec10 tx=0x7f6a9c00c490 comp rx=0 tx=0).stop 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 shutdown_connections 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8006f430 0x7f6a80071940 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6aa40cc930 0x7f6aa40c53a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6aa40cbd40 0x7f6aa4147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 --2- 172.21.15.81:0/4245302958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6aa40c6de0 0x7f6aa40ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 >> 172.21.15.81:0/4245302958 conn(0x7f6aa4068ab0 msgr2=0x7f6aa4068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 shutdown_connections 2024-05-27T06:57:30.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:30.054+0000 7f6aa8c00640 1 -- 172.21.15.81:0/4245302958 wait complete. 2024-05-27T06:57:30.067 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:57:31.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~87s 2024-05-27T06:57:32.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:32.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.234+0000 7f6a24000640 1 Processor -- start 2024-05-27T06:57:32.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.234+0000 7f6a24000640 1 -- start start 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a24000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 0x7f6a1c0c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a24000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a1c15db00 con 0x7f6a1c0c6fd0 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a24000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a1c073860 con 0x7f6a1c1546f0 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a24000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6a1c15e6f0 con 0x7f6a1c15ac20 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a20c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6a1c0c6fd0 0x7f6a1c15a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:48138/0 (socket says 172.21.15.81:48138) 2024-05-27T06:57:32.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a20c00640 1 -- 172.21.15.81:0/1739738806 learned_addr learned my addr 172.21.15.81:0/1739738806 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a22000640 1 --2- 172.21.15.81:0/1739738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 0x7f6a1c0c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579073637 0 0) 0x7f6a1c15db00 con 0x7f6a1c0c6fd0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a04003530 con 0x7f6a1c0c6fd0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261014224 0 0) 0x7f6a1c073860 con 0x7f6a1c1546f0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a1c15db00 con 0x7f6a1c1546f0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 873946033 0 0) 0x7f6a04003530 con 0x7f6a1c0c6fd0 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a22000640 1 -- 172.21.15.81:0/1739738806 >> v1:172.21.15.106:6789/0 conn(0x7f6a1c1546f0 legacy=0x7f6a1c072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a22000640 1 -- 172.21.15.81:0/1739738806 >> v1:172.21.15.81:6789/0 conn(0x7f6a1c0c6fd0 legacy=0x7f6a1c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.238+0000 7f6a22000640 1 -- 172.21.15.81:0/1739738806 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1c15ddc0 con 0x7f6a1c15ac20 2024-05-27T06:57:32.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a22000640 1 --2- 172.21.15.81:0/1739738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 0x7f6a1c0c73e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009dc0 tx=0x7f6a1401f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72bbea6d53498a16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:32.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a140b5070 con 0x7f6a1c15ac20 2024-05-27T06:57:32.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a140a8c30 con 0x7f6a1c15ac20 2024-05-27T06:57:32.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a1be00640 1 -- 172.21.15.81:0/1739738806 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a08001700 con 0x7f6a1c15ac20 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 -- 172.21.15.81:0/1739738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 msgr2=0x7f6a1c0c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 --2- 172.21.15.81:0/1739738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 0x7f6a1c0c73e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009dc0 tx=0x7f6a1401f0a0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 -- 172.21.15.81:0/1739738806 shutdown_connections 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 --2- 172.21.15.81:0/1739738806 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c15ac20 0x7f6a1c0c73e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 -- 172.21.15.81:0/1739738806 >> 172.21.15.81:0/1739738806 conn(0x7f6a1c070700 msgr2=0x7f6a1c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 -- 172.21.15.81:0/1739738806 shutdown_connections 2024-05-27T06:57:32.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.242+0000 7f6a24000640 1 -- 172.21.15.81:0/1739738806 wait complete. 2024-05-27T06:57:32.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 Processor -- start 2024-05-27T06:57:32.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- start start 2024-05-27T06:57:32.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 0x7f6a1c1f1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c1546f0 0x7f6a1c1588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c15e380 con 0x7f6a1c15ac20 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c0755b0 con 0x7f6a1c0c6fd0 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6a1c154b00 con 0x7f6a1c1546f0 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36544/0 (socket says 172.21.15.81:36544) 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 -- 172.21.15.81:0/1666482627 learned_addr learned my addr 172.21.15.81:0/1666482627 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a20c00640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c1546f0 0x7f6a1c1588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a21600640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 0x7f6a1c1f1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 -- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c1546f0 msgr2=0x7f6a1c1588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c1546f0 0x7f6a1c1588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 -- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 msgr2=0x7f6a1c1f1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 0x7f6a1c1f1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 -- 172.21.15.81:0/1666482627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a1c158e00 con 0x7f6a1c15ac20 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a21600640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 0x7f6a1c1f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a22000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6a1401d750 tx=0x7f6a1401be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:32.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a140b5050 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a140c1070 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14009450 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1c159e80 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.246+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a1c1553c0 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a1c154e50 con 0x7f6a1c15ac20 2024-05-27T06:57:32.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6a140bc670 con 0x7f6a1c15ac20 2024-05-27T06:57:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a19600640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 0x7f69fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6a1412dc10 con 0x7f6a1c15ac20 2024-05-27T06:57:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a21600640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 0x7f69fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:32.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.250+0000 7f6a21600640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 0x7f69fc071910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6a10001130 tx=0x7f6a10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:32.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.254+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6a14018c00 con 0x7f6a1c15ac20 2024-05-27T06:57:32.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.442+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a1c06a560 con 0x7f6a1c15ac20 2024-05-27T06:57:32.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a19600640 1 -- 172.21.15.81:0/1666482627 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6a140fa900 con 0x7f6a1c15ac20 2024-05-27T06:57:32.436 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:32.436 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:32.436 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:32.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 msgr2=0x7f69fc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 0x7f69fc071910 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6a10001130 tx=0x7f6a10009040 comp rx=0 tx=0).stop 2024-05-27T06:57:32.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 msgr2=0x7f6a1c0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:32.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6a1401d750 tx=0x7f6a1401be40 comp rx=0 tx=0).stop 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 shutdown_connections 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6a1c1546f0 0x7f6a1c1588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f69fc06f400 0x7f69fc071910 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1c15ac20 0x7f6a1c0ccca0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 --2- 172.21.15.81:0/1666482627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6a1c0c6fd0 0x7f6a1c1f1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 >> 172.21.15.81:0/1666482627 conn(0x7f6a1c070700 msgr2=0x7f6a1c1515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 shutdown_connections 2024-05-27T06:57:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:32.446+0000 7f6a24000640 1 -- 172.21.15.81:0/1666482627 wait complete. 2024-05-27T06:57:32.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:34.459 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:34.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 Processor -- start 2024-05-27T06:57:34.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 -- start start 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1be0156920 0x7f1be00ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1be00d00d0 0x7f1be0166f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be00ca180 con 0x7f1be00cfcc0 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1be00ca700 con 0x7f1be0156920 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdfe00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1be006e980 con 0x7f1be00d00d0 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1be0156920 0x7f1be00ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bde000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1be00cfcc0 0x7f1be0160240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:48148/0 (socket says 172.21.15.81:48148) 2024-05-27T06:57:34.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bde000640 1 -- 172.21.15.81:0/2476190879 learned_addr learned my addr 172.21.15.81:0/2476190879 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:34.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdea00640 1 -- 172.21.15.81:0/2476190879 >> v2:172.21.15.106:3300/0 conn(0x7f1be00d00d0 msgr2=0x7f1be0166f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:34.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdea00640 1 --2- 172.21.15.81:0/2476190879 >> v2:172.21.15.106:3300/0 conn(0x7f1be00d00d0 0x7f1be0166f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdea00640 1 -- 172.21.15.81:0/2476190879 >> v1:172.21.15.81:6789/0 conn(0x7f1be00cfcc0 legacy=0x7f1be0160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdea00640 1 -- 172.21.15.81:0/2476190879 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be00ca420 con 0x7f1be0156920 2024-05-27T06:57:34.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.630+0000 7f1bdd600640 1 -- 172.21.15.81:0/2476190879 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539781496 0 0) 0x7f1be00ca180 con 0x7f1be00cfcc0 2024-05-27T06:57:34.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdea00640 1 --2- 172.21.15.81:0/2476190879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be0156920 0x7f1be00ccbd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1be015d760 tx=0x7f1bd001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfed16e60d440850 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdd600640 1 -- 172.21.15.81:0/2476190879 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00b0070 con 0x7f1be0156920 2024-05-27T06:57:34.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdd600640 1 -- 172.21.15.81:0/2476190879 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd00ab600 con 0x7f1be0156920 2024-05-27T06:57:34.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdd600640 1 -- 172.21.15.81:0/2476190879 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00ab9d0 con 0x7f1be0156920 2024-05-27T06:57:34.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 -- 172.21.15.81:0/2476190879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be0156920 msgr2=0x7f1be00ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/2476190879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be0156920 0x7f1be00ccbd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1be015d760 tx=0x7f1bd001c3f0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 -- 172.21.15.81:0/2476190879 shutdown_connections 2024-05-27T06:57:34.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/2476190879 >> v2:172.21.15.106:3300/0 conn(0x7f1be00d00d0 0x7f1be0166f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/2476190879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be0156920 0x7f1be00ccbd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 -- 172.21.15.81:0/2476190879 >> 172.21.15.81:0/2476190879 conn(0x7f1be0068ab0 msgr2=0x7f1be0068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:34.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 -- 172.21.15.81:0/2476190879 shutdown_connections 2024-05-27T06:57:34.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.634+0000 7f1bdfe00640 1 -- 172.21.15.81:0/2476190879 wait complete. 2024-05-27T06:57:34.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 Processor -- start 2024-05-27T06:57:34.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- start start 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1be00d00d0 0x7f1be0152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 0x7f1be014cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0072b80 con 0x7f1be00cfcc0 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1be0156d30 con 0x7f1be0156920 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1be00d04e0 con 0x7f1be00d00d0 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36560/0 (socket says 172.21.15.81:36560) 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 -- 172.21.15.81:0/3600002014 learned_addr learned my addr 172.21.15.81:0/3600002014 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bde000640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1be00d00d0 0x7f1be0152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdf400640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 0x7f1be014cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 -- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1be00d00d0 msgr2=0x7f1be0152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1be00d00d0 0x7f1be0152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 -- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 msgr2=0x7f1be014cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 0x7f1be014cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 -- 172.21.15.81:0/3600002014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be00ce000 con 0x7f1be00cfcc0 2024-05-27T06:57:34.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdf400640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 0x7f1be014cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:57:34.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdea00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00ae040 tx=0x7f1bd001ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00b0050 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd00bc070 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00b5a70 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be00ce680 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.638+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be01f96d0 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.642+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be0050870 con 0x7f1be00cfcc0 2024-05-27T06:57:34.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.642+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1bd00ab480 con 0x7f1be00cfcc0 2024-05-27T06:57:34.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.642+0000 7f1bd7400640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:34.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.642+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f1bd0129b70 con 0x7f1be00cfcc0 2024-05-27T06:57:34.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.642+0000 7f1bde000640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:34.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.646+0000 7f1bde000640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 0x7f1bc0071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1be0072570 tx=0x7f1bcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:34.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.650+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1bd00c6020 con 0x7f1be00cfcc0 2024-05-27T06:57:34.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:57:34.712 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:57:34.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.842+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1be006de60 con 0x7f1be00cfcc0 2024-05-27T06:57:34.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.842+0000 7f1bd7400640 1 -- 172.21.15.81:0/3600002014 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f1bd0016070 con 0x7f1be00cfcc0 2024-05-27T06:57:34.835 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:34.835 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:34.836 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 msgr2=0x7f1bc0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 0x7f1bc0071910 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1be0072570 tx=0x7f1bcc009040 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 msgr2=0x7f1be00cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1bd00ae040 tx=0x7f1bd001ac30 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 shutdown_connections 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1be00d00d0 0x7f1be0152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1bc006f400 0x7f1bc0071910 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1be0156920 0x7f1be014cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 --2- 172.21.15.81:0/3600002014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00cfcc0 0x7f1be00cdac0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 >> 172.21.15.81:0/3600002014 conn(0x7f1be0068ab0 msgr2=0x7f1be0153820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 shutdown_connections 2024-05-27T06:57:34.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:34.846+0000 7f1bdfe00640 1 -- 172.21.15.81:0/3600002014 wait complete. 2024-05-27T06:57:34.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:36.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:37.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.026+0000 7f6ab5800640 1 Processor -- start 2024-05-27T06:57:37.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.026+0000 7f6ab5800640 1 -- start start 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ab015fd20 0x7f6ab0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00c89a0 0x7f6ab0072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6ab00ca900 0x7f6ab00cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab00c9070 con 0x7f6ab015fd20 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab0074890 con 0x7f6ab00ca900 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6ab5800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab0073d00 con 0x7f6ab00c89a0 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ab015fd20 0x7f6ab0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ab015fd20 0x7f6ab0156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47558/0 (socket says 172.21.15.81:47558) 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 -- 172.21.15.81:0/740616212 learned_addr learned my addr 172.21.15.81:0/740616212 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaf400640 1 --2- 172.21.15.81:0/740616212 >> v2:172.21.15.106:3300/0 conn(0x7f6ab00ca900 0x7f6ab00cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aa7e00640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00c89a0 0x7f6ab0072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 -- 172.21.15.81:0/740616212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00c89a0 msgr2=0x7f6ab0072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:37.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00c89a0 0x7f6ab0072c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 -- 172.21.15.81:0/740616212 >> v2:172.21.15.106:3300/0 conn(0x7f6ab00ca900 msgr2=0x7f6ab00cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:37.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 --2- 172.21.15.81:0/740616212 >> v2:172.21.15.106:3300/0 conn(0x7f6ab00ca900 0x7f6ab00cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.030+0000 7f6aaea00640 1 -- 172.21.15.81:0/740616212 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab0160130 con 0x7f6ab015fd20 2024-05-27T06:57:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6aaea00640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0156a30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0071340 tx=0x7f6aa001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbc707c5a0c11f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6aae000640 1 -- 172.21.15.81:0/740616212 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00b0070 con 0x7f6ab015fd20 2024-05-27T06:57:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6aae000640 1 -- 172.21.15.81:0/740616212 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa00ab600 con 0x7f6ab015fd20 2024-05-27T06:57:37.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6aae000640 1 -- 172.21.15.81:0/740616212 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00a4e60 con 0x7f6ab015fd20 2024-05-27T06:57:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- 172.21.15.81:0/740616212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 msgr2=0x7f6ab0156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0156a30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0071340 tx=0x7f6aa001c520 comp rx=0 tx=0).stop 2024-05-27T06:57:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- 172.21.15.81:0/740616212 shutdown_connections 2024-05-27T06:57:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 --2- 172.21.15.81:0/740616212 >> v2:172.21.15.106:3300/0 conn(0x7f6ab00ca900 0x7f6ab00cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00c89a0 0x7f6ab0072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 --2- 172.21.15.81:0/740616212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0156a30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- 172.21.15.81:0/740616212 >> 172.21.15.81:0/740616212 conn(0x7f6ab006f230 msgr2=0x7f6ab0070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- 172.21.15.81:0/740616212 shutdown_connections 2024-05-27T06:57:37.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- 172.21.15.81:0/740616212 wait complete. 2024-05-27T06:57:37.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 Processor -- start 2024-05-27T06:57:37.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.034+0000 7f6ab5800640 1 -- start start 2024-05-27T06:57:37.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab00c89a0 0x7f6ab014f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00ca900 0x7f6ab01f9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab00c94c0 con 0x7f6ab015fd20 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab00cb250 con 0x7f6ab00c89a0 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab00c9760 con 0x7f6ab00ca900 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47572/0 (socket says 172.21.15.81:47572) 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 -- 172.21.15.81:0/3678850856 learned_addr learned my addr 172.21.15.81:0/3678850856 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaea00640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab00c89a0 0x7f6ab014f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aa7e00640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00ca900 0x7f6ab01f9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 -- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00ca900 msgr2=0x7f6ab01f9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00ca900 0x7f6ab01f9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 -- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab00c89a0 msgr2=0x7f6ab014f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab00c89a0 0x7f6ab014f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 -- 172.21.15.81:0/3678850856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab01f9840 con 0x7f6ab015fd20 2024-05-27T06:57:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aaf400640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8007ec0 tx=0x7f6aa800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8014420 con 0x7f6ab015fd20 2024-05-27T06:57:37.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa8019070 con 0x7f6ab015fd20 2024-05-27T06:57:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab0159290 con 0x7f6ab015fd20 2024-05-27T06:57:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8014820 con 0x7f6ab015fd20 2024-05-27T06:57:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab0159850 con 0x7f6ab015fd20 2024-05-27T06:57:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.038+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab006a560 con 0x7f6ab015fd20 2024-05-27T06:57:37.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.042+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6aa80149c0 con 0x7f6ab015fd20 2024-05-27T06:57:37.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.042+0000 7f6aa6a00640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 0x7f6a8c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:37.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.042+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6aa808fb60 con 0x7f6ab015fd20 2024-05-27T06:57:37.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.042+0000 7f6aaea00640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 0x7f6a8c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:37.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.042+0000 7f6aaea00640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 0x7f6a8c071910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00ad000 tx=0x7f6aa0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:37.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.046+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6aa805c850 con 0x7f6ab015fd20 2024-05-27T06:57:37.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.234+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ab015a150 con 0x7f6ab015fd20 2024-05-27T06:57:37.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.234+0000 7f6aa6a00640 1 -- 172.21.15.81:0/3678850856 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6aa805c150 con 0x7f6ab015fd20 2024-05-27T06:57:37.227 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:37.227 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:37.227 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 msgr2=0x7f6a8c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 0x7f6a8c071910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00ad000 tx=0x7f6aa0008040 comp rx=0 tx=0).stop 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 msgr2=0x7f6ab0157020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8007ec0 tx=0x7f6aa800e4b0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 shutdown_connections 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6ab00ca900 0x7f6ab01f9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6a8c06f400 0x7f6a8c071910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab015fd20 0x7f6ab0157020 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 --2- 172.21.15.81:0/3678850856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab00c89a0 0x7f6ab014f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:37.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 >> 172.21.15.81:0/3678850856 conn(0x7f6ab006f230 msgr2=0x7f6ab015e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:37.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 shutdown_connections 2024-05-27T06:57:37.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:37.238+0000 7f6ab5800640 1 -- 172.21.15.81:0/3678850856 wait complete. 2024-05-27T06:57:37.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:38.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~94s 2024-05-27T06:57:39.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:39.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 Processor -- start 2024-05-27T06:57:39.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 -- start start 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f82300cfd60 0x7f82300d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82300ca950 con 0x7f82300ca380 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82300cb670 con 0x7f823015fd90 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8237800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f82300cb420 con 0x7f82300cfd60 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8234e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f82300ca380 0x7f82301566e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55378/0 (socket says 172.21.15.81:55378) 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f8234e00640 1 -- 172.21.15.81:0/1451404981 learned_addr learned my addr 172.21.15.81:0/1451404981 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822fe00640 1 --2- 172.21.15.81:0/1451404981 >> v2:172.21.15.106:3300/0 conn(0x7f82300cfd60 0x7f82300d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3832350360 0 0) 0x7f82300ca950 con 0x7f82300ca380 2024-05-27T06:57:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8214003530 con 0x7f82300ca380 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3724363972 0 0) 0x7f8214003530 con 0x7f82300ca380 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 >> v2:172.21.15.106:3300/0 conn(0x7f82300cfd60 msgr2=0x7f82300d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 --2- 172.21.15.81:0/1451404981 >> v2:172.21.15.106:3300/0 conn(0x7f82300cfd60 0x7f82300d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 >> v1:172.21.15.106:6789/0 conn(0x7f823015fd90 legacy=0x7f82301601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82300cb950 con 0x7f82300ca380 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f8220001a80 con 0x7f82300ca380 2024-05-27T06:57:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 >> v1:172.21.15.81:6789/0 conn(0x7f82300ca380 legacy=0x7f82301566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8214003a60 0x7f8214003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.418+0000 7f822f400640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 0x7f8214004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214005260 0x7f821400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82300cb420 con 0x7f8214004810 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f821400c1f0 con 0x7f8214003a60 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f821400c3f0 con 0x7f8214005260 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 0x7f8214004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f8234e00640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8214003a60 0x7f8214003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f8235800640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214005260 0x7f821400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 -- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214005260 msgr2=0x7f821400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214005260 0x7f821400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 -- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8214003a60 msgr2=0x7f8214003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8214003a60 0x7f8214003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 -- 172.21.15.81:0/1451404981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8214003530 con 0x7f8214004810 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f8235800640 1 -- 172.21.15.81:0/1451404981 reap_dead start 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822fe00640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 0x7f8214004c20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f821c009ae0 tx=0x7f821c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2d1b5a0e0c07ed server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0b2070 con 0x7f8214004810 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c0a5660 con 0x7f8214004810 2024-05-27T06:57:39.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.422+0000 7f822f400640 1 -- 172.21.15.81:0/1451404981 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0096b0 con 0x7f8214004810 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 msgr2=0x7f8214004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 0x7f8214004c20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f821c009ae0 tx=0x7f821c01ca70 comp rx=0 tx=0).stop 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- 172.21.15.81:0/1451404981 shutdown_connections 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 --2- 172.21.15.81:0/1451404981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8214004810 0x7f8214004c20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- 172.21.15.81:0/1451404981 >> 172.21.15.81:0/1451404981 conn(0x7f8230068ae0 msgr2=0x7f8230068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:39.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- 172.21.15.81:0/1451404981 shutdown_connections 2024-05-27T06:57:39.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- 172.21.15.81:0/1451404981 wait complete. 2024-05-27T06:57:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 Processor -- start 2024-05-27T06:57:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.426+0000 7f8237800640 1 -- start start 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214004810 0x7f82301559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8230153f90 0x7f8230150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8230072e40 con 0x7f82300c8280 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f823006d8a0 con 0x7f8230153f90 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f82300d0170 con 0x7f8214004810 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47584/0 (socket says 172.21.15.81:47584) 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 -- 172.21.15.81:0/3753646497 learned_addr learned my addr 172.21.15.81:0/3753646497 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8234e00640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214004810 0x7f82301559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f822fe00640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8230153f90 0x7f8230150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 -- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214004810 msgr2=0x7f82301559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214004810 0x7f82301559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 -- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8230153f90 msgr2=0x7f8230150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8230153f90 0x7f8230150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 -- 172.21.15.81:0/3753646497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8230156280 con 0x7f82300c8280 2024-05-27T06:57:39.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8235800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8228000c10 tx=0x7f822800e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228019070 con 0x7f82300c8280 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82280052e0 con 0x7f82300c8280 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82301e7ae0 con 0x7f82300c8280 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228003840 con 0x7f82300c8280 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82301e80a0 con 0x7f82300c8280 2024-05-27T06:57:39.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823006a560 con 0x7f82300c8280 2024-05-27T06:57:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.430+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f822801c070 con 0x7f82300c8280 2024-05-27T06:57:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.434+0000 7f822cc00640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 0x7f8210071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.434+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f822808e8b0 con 0x7f82300c8280 2024-05-27T06:57:39.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.434+0000 7f8234e00640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 0x7f8210071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:39.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.434+0000 7f8234e00640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 0x7f8210071940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f82200015b0 tx=0x7f8220002200 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:39.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.438+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f822805af50 con 0x7f82300c8280 2024-05-27T06:57:39.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.630+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f823015d3b0 con 0x7f82300c8280 2024-05-27T06:57:39.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.630+0000 7f822cc00640 1 -- 172.21.15.81:0/3753646497 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f822805f250 con 0x7f82300c8280 2024-05-27T06:57:39.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:39.623 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:39.624 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:39.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 msgr2=0x7f8210071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 0x7f8210071940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f82200015b0 tx=0x7f8220002200 comp rx=0 tx=0).stop 2024-05-27T06:57:39.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 msgr2=0x7f82300c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:39.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8228000c10 tx=0x7f822800e820 comp rx=0 tx=0).stop 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 shutdown_connections 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f821006f430 0x7f8210071940 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82300c8280 0x7f82300c8690 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8230153f90 0x7f8230150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 --2- 172.21.15.81:0/3753646497 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8214004810 0x7f82301559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 >> 172.21.15.81:0/3753646497 conn(0x7f8230068ae0 msgr2=0x7f82300ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 shutdown_connections 2024-05-27T06:57:39.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:39.634+0000 7f8237800640 1 -- 172.21.15.81:0/3753646497 wait complete. 2024-05-27T06:57:39.645 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:41.646 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 Processor -- start 2024-05-27T06:57:41.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 -- start start 2024-05-27T06:57:41.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c156820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9c15cb60 con 0x7f8d9c0cebf0 2024-05-27T06:57:41.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f8d9c15ce40 con 0x7f8d9c160370 2024-05-27T06:57:41.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da2400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c15d120 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8d9be00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8d9be00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:35588/0 (socket says 172.21.15.81:35588) 2024-05-27T06:57:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8d9be00640 1 -- 172.21.15.81:0/2586694520 learned_addr learned my addr 172.21.15.81:0/2586694520 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:41.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da1000640 1 -- 172.21.15.81:0/2586694520 >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 msgr2=0x7f8d9c160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:41.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da1000640 1 --2- 172.21.15.81:0/2586694520 >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da1000640 1 -- 172.21.15.81:0/2586694520 >> v1:172.21.15.106:6789/0 conn(0x7f8d9c0cebf0 legacy=0x7f8d9c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8da1000640 1 -- 172.21.15.81:0/2586694520 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c15d400 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.814+0000 7f8d9be00640 1 --2- 172.21.15.81:0/2586694520 >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:57:41.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da1000640 1 --2- 172.21.15.81:0/2586694520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c156820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097c0 tx=0x7f8d9001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45a9a80b7409c3e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8d9b400640 1 -- 172.21.15.81:0/2586694520 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d900b5070 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8d9b400640 1 -- 172.21.15.81:0/2586694520 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d900a8990 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8d9b400640 1 -- 172.21.15.81:0/2586694520 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d900a8d60 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 -- 172.21.15.81:0/2586694520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 msgr2=0x7f8d9c156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 --2- 172.21.15.81:0/2586694520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c156820 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097c0 tx=0x7f8d9001ec90 comp rx=0 tx=0).stop 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 -- 172.21.15.81:0/2586694520 shutdown_connections 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 --2- 172.21.15.81:0/2586694520 >> v2:172.21.15.106:3300/0 conn(0x7f8d9c160370 0x7f8d9c160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 --2- 172.21.15.81:0/2586694520 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c156820 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 -- 172.21.15.81:0/2586694520 >> 172.21.15.81:0/2586694520 conn(0x7f8d9c071390 msgr2=0x7f8d9c0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:41.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 -- 172.21.15.81:0/2586694520 shutdown_connections 2024-05-27T06:57:41.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.818+0000 7f8da2400640 1 -- 172.21.15.81:0/2586694520 wait complete. 2024-05-27T06:57:41.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 Processor -- start 2024-05-27T06:57:41.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 -- start start 2024-05-27T06:57:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c15f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d9c0cebf0 0x7f8d9c1e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c156ee0 con 0x7f8d9c160370 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c156d60 con 0x7f8d9c0cebf0 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da2400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c15c720 con 0x7f8d9c0c96b0 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47586/0 (socket says 172.21.15.81:47586) 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da1a00640 1 -- 172.21.15.81:0/4065333181 learned_addr learned my addr 172.21.15.81:0/4065333181 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8d9be00640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d9c0cebf0 0x7f8d9c1e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.822+0000 7f8da1000640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c15f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 -- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 msgr2=0x7f8d9c15f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c15f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 -- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d9c0cebf0 msgr2=0x7f8d9c1e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d9c0cebf0 0x7f8d9c1e68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:41.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 -- 172.21.15.81:0/4065333181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c15f820 con 0x7f8d9c160370 2024-05-27T06:57:41.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da1a00640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400c730 tx=0x7f8d9400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94003b80 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d94003d20 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9c155730 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400aa70 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9c0c6470 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9c050870 con 0x7f8d9c160370 2024-05-27T06:57:41.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.826+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8d9400ac10 con 0x7f8d9c160370 2024-05-27T06:57:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.830+0000 7f8d98c00640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 0x7f8d7c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.830+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f8d9408f2f0 con 0x7f8d9c160370 2024-05-27T06:57:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.830+0000 7f8da1000640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 0x7f8d7c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:41.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.830+0000 7f8da1000640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 0x7f8d7c071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b2000 tx=0x7f8d90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:41.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:41.834+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8d9405b910 con 0x7f8d9c160370 2024-05-27T06:57:42.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d9c0c8360 con 0x7f8d9c160370 2024-05-27T06:57:42.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8d98c00640 1 -- 172.21.15.81:0/4065333181 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f8d9405fc10 con 0x7f8d9c160370 2024-05-27T06:57:42.017 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:42.017 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:42.017 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 msgr2=0x7f8d7c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 0x7f8d7c071910 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b2000 tx=0x7f8d90008040 comp rx=0 tx=0).stop 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 msgr2=0x7f8d9c151090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400c730 tx=0x7f8d9400cbf0 comp rx=0 tx=0).stop 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 shutdown_connections 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8d7c06f400 0x7f8d7c071910 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d9c160370 0x7f8d9c151090 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:42.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8d9c0cebf0 0x7f8d9c1e68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:42.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 --2- 172.21.15.81:0/4065333181 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8d9c0c96b0 0x7f8d9c15f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:42.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 >> 172.21.15.81:0/4065333181 conn(0x7f8d9c071390 msgr2=0x7f8d9c15a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:42.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 shutdown_connections 2024-05-27T06:57:42.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:42.026+0000 7f8da2400640 1 -- 172.21.15.81:0/4065333181 wait complete. 2024-05-27T06:57:42.040 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:57:44.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.206+0000 7fa6f5e00640 1 Processor -- start 2024-05-27T06:57:44.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.206+0000 7fa6f5e00640 1 -- start start 2024-05-27T06:57:44.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6f5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa6f0072430 0x7fa6f0159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6f5e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa6f0161cd0 0x7fa6f01640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6f5e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f0072840 con 0x7fa6f015a040 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6f5e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f00c5040 con 0x7fa6f0072430 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6f5e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f00c75c0 con 0x7fa6f0161cd0 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa6f0072430 0x7fa6f0159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa6f0072430 0x7fa6f0159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47588/0 (socket says 172.21.15.81:47588) 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 -- 172.21.15.81:0/1671666902 learned_addr learned my addr 172.21.15.81:0/1671666902 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 -- 172.21.15.81:0/1671666902 >> v2:172.21.15.106:3300/0 conn(0x7fa6f0161cd0 msgr2=0x7fa6f01640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 --2- 172.21.15.81:0/1671666902 >> v2:172.21.15.106:3300/0 conn(0x7fa6f0161cd0 0x7fa6f01640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 -- 172.21.15.81:0/1671666902 >> v1:172.21.15.106:6789/0 conn(0x7fa6f015a040 legacy=0x7fa6f00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:44.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.210+0000 7fa6eea00640 1 -- 172.21.15.81:0/1671666902 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f00c5270 con 0x7fa6f0072430 2024-05-27T06:57:44.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6eea00640 1 --2- 172.21.15.81:0/1671666902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0072430 0x7fa6f0159b00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00c9d90 tx=0x7fa6e001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c2fc14af5e67c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6ee000640 1 -- 172.21.15.81:0/1671666902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00a5670 con 0x7fa6f0072430 2024-05-27T06:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6ee000640 1 -- 172.21.15.81:0/1671666902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e00a5810 con 0x7fa6f0072430 2024-05-27T06:57:44.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6ee000640 1 -- 172.21.15.81:0/1671666902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00a5be0 con 0x7fa6f0072430 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 -- 172.21.15.81:0/1671666902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0072430 msgr2=0x7fa6f0159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/1671666902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0072430 0x7fa6f0159b00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa6f00c9d90 tx=0x7fa6e001c9f0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 -- 172.21.15.81:0/1671666902 shutdown_connections 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/1671666902 >> v2:172.21.15.106:3300/0 conn(0x7fa6f0161cd0 0x7fa6f01640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/1671666902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0072430 0x7fa6f0159b00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 -- 172.21.15.81:0/1671666902 >> 172.21.15.81:0/1671666902 conn(0x7fa6f0070700 msgr2=0x7fa6f00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:44.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 -- 172.21.15.81:0/1671666902 shutdown_connections 2024-05-27T06:57:44.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.214+0000 7fa6f5e00640 1 -- 172.21.15.81:0/1671666902 wait complete. 2024-05-27T06:57:44.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 Processor -- start 2024-05-27T06:57:44.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- start start 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 0x7fa6f01518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6f015a040 0x7fa6f0149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f00ce6f0 con 0x7fa6f0161cd0 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f00c7dd0 con 0x7fa6f0072430 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f00c5610 con 0x7fa6f015a040 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47594/0 (socket says 172.21.15.81:47594) 2024-05-27T06:57:44.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 -- 172.21.15.81:0/3290867969 learned_addr learned my addr 172.21.15.81:0/3290867969 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6eea00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 0x7fa6f01518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6e6000640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6f015a040 0x7fa6f0149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 -- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6f015a040 msgr2=0x7fa6f0149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6f015a040 0x7fa6f0149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 -- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 msgr2=0x7fa6f01518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 0x7fa6f01518d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 -- 172.21.15.81:0/3290867969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f014a290 con 0x7fa6f0161cd0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6eea00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 0x7fa6f01518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6ef400640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800b410 tx=0x7fa6e800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8015020 con 0x7fa6f0161cd0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e8011830 con 0x7fa6f0161cd0 2024-05-27T06:57:44.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f00c4e00 con 0x7fa6f0161cd0 2024-05-27T06:57:44.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8011c20 con 0x7fa6f0161cd0 2024-05-27T06:57:44.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.218+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f01f9950 con 0x7fa6f0161cd0 2024-05-27T06:57:44.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f00ce1c0 con 0x7fa6f0161cd0 2024-05-27T06:57:44.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa6e8018020 con 0x7fa6f0161cd0 2024-05-27T06:57:44.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6e7e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 0x7fa6cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:44.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa6e808f420 con 0x7fa6f0161cd0 2024-05-27T06:57:44.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6eea00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 0x7fa6cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:44.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.222+0000 7fa6eea00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 0x7fa6cc071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00ae000 tx=0x7fa6e00a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:44.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.226+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa6e801f020 con 0x7fa6f0161cd0 2024-05-27T06:57:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6f006a560 con 0x7fa6f0161cd0 2024-05-27T06:57:44.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6e7e00640 1 -- 172.21.15.81:0/3290867969 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fa6e805ba30 con 0x7fa6f0161cd0 2024-05-27T06:57:44.409 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:44.409 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:44.409 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:44.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 msgr2=0x7fa6cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 0x7fa6cc071910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00ae000 tx=0x7fa6e00a7070 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 msgr2=0x7fa6f0147630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800b410 tx=0x7fa6e800b8d0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 shutdown_connections 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa6f015a040 0x7fa6f0149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa6cc06f400 0x7fa6cc071910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6f0161cd0 0x7fa6f0147630 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 --2- 172.21.15.81:0/3290867969 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6f0072430 0x7fa6f01518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 >> 172.21.15.81:0/3290867969 conn(0x7fa6f0070700 msgr2=0x7fa6f0148b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 shutdown_connections 2024-05-27T06:57:44.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:44.418+0000 7fa6f5e00640 1 -- 172.21.15.81:0/3290867969 wait complete. 2024-05-27T06:57:44.430 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:45.034 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~101s 2024-05-27T06:57:46.431 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:46.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 Processor -- start 2024-05-27T06:57:46.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 -- start start 2024-05-27T06:57:46.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f6998160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69980c99f0 con 0x7f6998156bc0 2024-05-27T06:57:46.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69981607c0 con 0x7f69981650d0 2024-05-27T06:57:46.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f699dc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6998156fd0 con 0x7f699815fe70 2024-05-27T06:57:46.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6998156bc0 0x7f69980cc2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33582/0 (socket says 172.21.15.81:33582) 2024-05-27T06:57:46.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996a00640 1 -- 172.21.15.81:0/4245480053 learned_addr learned my addr 172.21.15.81:0/4245480053 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:46.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f698fe00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f6998160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241149788 0 0) 0x7f69980c99f0 con 0x7f6998156bc0 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f697c003530 con 0x7f6998156bc0 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 896712974 0 0) 0x7f697c003530 con 0x7f6998156bc0 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 msgr2=0x7f6998160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f6998160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 >> v1:172.21.15.106:6789/0 conn(0x7f69981650d0 legacy=0x7f69981674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699815c980 con 0x7f6998156bc0 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f6988001700 con 0x7f6998156bc0 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 >> v1:172.21.15.81:6789/0 conn(0x7f6998156bc0 legacy=0x7f69980cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f697c003a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f697c004010 0x7f697c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f697c004d50 0x7f697c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6998156fd0 con 0x7f697c004010 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f697c00bca0 con 0x7f697c004d50 2024-05-27T06:57:46.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.602+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f697c00bea0 con 0x7f699815fe70 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f6996a00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f697c003a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f697c004010 0x7f697c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f6997400640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f697c004d50 0x7f697c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 -- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 msgr2=0x7f697c003a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f699815fe70 0x7f697c003a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 -- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f697c004d50 msgr2=0x7f697c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f697c004d50 0x7f697c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 -- 172.21.15.81:0/4245480053 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c003530 con 0x7f697c004010 2024-05-27T06:57:46.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f698fe00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f697c004010 0x7f697c004810 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6998071ad0 tx=0x7f698001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a3faf148d1e24d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:46.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800b2070 con 0x7f697c004010 2024-05-27T06:57:46.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69800a5620 con 0x7f697c004010 2024-05-27T06:57:46.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.606+0000 7f6996000640 1 -- 172.21.15.81:0/4245480053 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800096b0 con 0x7f697c004010 2024-05-27T06:57:46.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- 172.21.15.81:0/4245480053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f697c004010 msgr2=0x7f697c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 --2- 172.21.15.81:0/4245480053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f697c004010 0x7f697c004810 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6998071ad0 tx=0x7f698001ca30 comp rx=0 tx=0).stop 2024-05-27T06:57:46.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f6997400640 1 -- 172.21.15.81:0/4245480053 reap_dead start 2024-05-27T06:57:46.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- 172.21.15.81:0/4245480053 shutdown_connections 2024-05-27T06:57:46.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- 172.21.15.81:0/4245480053 >> 172.21.15.81:0/4245480053 conn(0x7f6998068cb0 msgr2=0x7f69980745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:46.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- 172.21.15.81:0/4245480053 shutdown_connections 2024-05-27T06:57:46.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- 172.21.15.81:0/4245480053 wait complete. 2024-05-27T06:57:46.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 Processor -- start 2024-05-27T06:57:46.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.610+0000 7f699dc00640 1 -- start start 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f697c004010 0x7f69981f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69980c7ae0 0x7f6998148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f699815d220 con 0x7f699814e170 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f699815cf40 con 0x7f69980c7ae0 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f699815d500 con 0x7f697c004010 2024-05-27T06:57:46.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33922/0 (socket says 172.21.15.81:33922) 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 -- 172.21.15.81:0/2689482489 learned_addr learned my addr 172.21.15.81:0/2689482489 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f6996a00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f697c004010 0x7f69981f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f6997400640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69980c7ae0 0x7f6998148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 -- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f697c004010 msgr2=0x7f69981f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f697c004010 0x7f69981f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 -- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69980c7ae0 msgr2=0x7f6998148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69980c7ae0 0x7f6998148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 -- 172.21.15.81:0/2689482489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69981f6030 con 0x7f699814e170 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698fe00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f698001c860 tx=0x7f697c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800bd070 con 0x7f699814e170 2024-05-27T06:57:46.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6980004030 con 0x7f699814e170 2024-05-27T06:57:46.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69981f6d00 con 0x7f699814e170 2024-05-27T06:57:46.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69800b2040 con 0x7f699814e170 2024-05-27T06:57:46.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69981f62a0 con 0x7f699814e170 2024-05-27T06:57:46.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699806a560 con 0x7f699814e170 2024-05-27T06:57:46.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.614+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f69800c0070 con 0x7f699814e170 2024-05-27T06:57:46.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.618+0000 7f698ea00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 0x7f6974071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:46.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.618+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6980129490 con 0x7f699814e170 2024-05-27T06:57:46.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.618+0000 7f6996a00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 0x7f6974071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:46.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.618+0000 7f6996a00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 0x7f6974071910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6988000fc0 tx=0x7f6988009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:46.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.622+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6980016740 con 0x7f699814e170 2024-05-27T06:57:46.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.814+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69980caaf0 con 0x7f699814e170 2024-05-27T06:57:46.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.814+0000 7f698ea00640 1 -- 172.21.15.81:0/2689482489 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f69800f5b30 con 0x7f699814e170 2024-05-27T06:57:46.807 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:46.807 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:46.807 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 msgr2=0x7f6974071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 0x7f6974071910 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6988000fc0 tx=0x7f6988009040 comp rx=0 tx=0).stop 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 msgr2=0x7f6998148200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f698001c860 tx=0x7f697c005840 comp rx=0 tx=0).stop 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 shutdown_connections 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f697406f400 0x7f6974071910 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f69980c7ae0 0x7f6998148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f699814e170 0x7f6998148200 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 --2- 172.21.15.81:0/2689482489 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f697c004010 0x7f69981f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 >> 172.21.15.81:0/2689482489 conn(0x7f6998068cb0 msgr2=0x7f699814ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 shutdown_connections 2024-05-27T06:57:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:46.818+0000 7f699dc00640 1 -- 172.21.15.81:0/2689482489 wait complete. 2024-05-27T06:57:46.829 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:48.829 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 Processor -- start 2024-05-27T06:57:48.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 -- start start 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4084154780 0x7f4084072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4084072a60 0x7f4084164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40840c9c70 con 0x7f40840c9630 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f40840c9a40 con 0x7f4084072a60 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f408415b0c0 con 0x7f4084154780 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4084072a60 0x7f4084164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4082a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f40840c9630 0x7f408415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33590/0 (socket says 172.21.15.81:33590) 2024-05-27T06:57:48.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4082a00640 1 -- 172.21.15.81:0/3354576168 learned_addr learned my addr 172.21.15.81:0/3354576168 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4082000640 1 --2- 172.21.15.81:0/3354576168 >> v2:172.21.15.106:3300/0 conn(0x7f4084154780 0x7f4084072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083400640 1 -- 172.21.15.81:0/3354576168 >> v2:172.21.15.106:3300/0 conn(0x7f4084154780 msgr2=0x7f4084072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083400640 1 --2- 172.21.15.81:0/3354576168 >> v2:172.21.15.106:3300/0 conn(0x7f4084154780 0x7f4084072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083400640 1 -- 172.21.15.81:0/3354576168 >> v1:172.21.15.81:6789/0 conn(0x7f40840c9630 legacy=0x7f408415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:48.998+0000 7f4083400640 1 -- 172.21.15.81:0/3354576168 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408415a4d0 con 0x7f4084072a60 2024-05-27T06:57:48.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083400640 1 --2- 172.21.15.81:0/3354576168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f4084164e50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f40840c9550 tx=0x7f407c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ea24962667ad7d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:48.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4081600640 1 -- 172.21.15.81:0/3354576168 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0b2070 con 0x7f4084072a60 2024-05-27T06:57:48.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4081600640 1 -- 172.21.15.81:0/3354576168 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407c0a5870 con 0x7f4084072a60 2024-05-27T06:57:48.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4081600640 1 -- 172.21.15.81:0/3354576168 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a5c40 con 0x7f4084072a60 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 -- 172.21.15.81:0/3354576168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 msgr2=0x7f4084164e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 --2- 172.21.15.81:0/3354576168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f4084164e50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f40840c9550 tx=0x7f407c01c8a0 comp rx=0 tx=0).stop 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 -- 172.21.15.81:0/3354576168 shutdown_connections 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 --2- 172.21.15.81:0/3354576168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f4084164e50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 --2- 172.21.15.81:0/3354576168 >> v2:172.21.15.106:3300/0 conn(0x7f4084154780 0x7f4084072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 -- 172.21.15.81:0/3354576168 >> 172.21.15.81:0/3354576168 conn(0x7f4084068880 msgr2=0x7f4084068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 -- 172.21.15.81:0/3354576168 shutdown_connections 2024-05-27T06:57:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.002+0000 7f4083e00640 1 -- 172.21.15.81:0/3354576168 wait complete. 2024-05-27T06:57:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 Processor -- start 2024-05-27T06:57:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 -- start start 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40840c9630 0x7f40841481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4084154780 0x7f4084150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40840cd280 con 0x7f4084072a60 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f408415ad50 con 0x7f4084154780 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f408415e1d0 con 0x7f40840c9630 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33932/0 (socket says 172.21.15.81:33932) 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 -- 172.21.15.81:0/4209460216 learned_addr learned my addr 172.21.15.81:0/4209460216 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4083400640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4084154780 0x7f4084150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082000640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40840c9630 0x7f40841481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 -- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40840c9630 msgr2=0x7f40841481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40840c9630 0x7f40841481a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 -- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4084154780 msgr2=0x7f4084150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4084154780 0x7f4084150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 -- 172.21.15.81:0/4209460216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408415d0e0 con 0x7f4084072a60 2024-05-27T06:57:49.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f4082a00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f407400c3f0 tx=0x7f407400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.006+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074017070 con 0x7f4084072a60 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4074005d10 con 0x7f4084072a60 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074007310 con 0x7f4084072a60 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40841f3cb0 con 0x7f4084072a60 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40841f3f70 con 0x7f4084072a60 2024-05-27T06:57:49.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4084050870 con 0x7f4084072a60 2024-05-27T06:57:49.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f407401a070 con 0x7f4084072a60 2024-05-27T06:57:49.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f407aa00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 0x7f4064071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:49.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f407408e670 con 0x7f4084072a60 2024-05-27T06:57:49.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.010+0000 7f4082000640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 0x7f4064071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:49.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.014+0000 7f4082000640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 0x7f4064071940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f40840c6800 tx=0x7f4070008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:49.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.018+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f407405ad10 con 0x7f4084072a60 2024-05-27T06:57:49.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.206+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40840ccf20 con 0x7f4084072a60 2024-05-27T06:57:49.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.206+0000 7f407aa00640 1 -- 172.21.15.81:0/4209460216 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f407405f010 con 0x7f4084072a60 2024-05-27T06:57:49.198 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:49.199 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:49.199 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:49.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 msgr2=0x7f4064071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:49.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 0x7f4064071940 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f40840c6800 tx=0x7f4070008040 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 msgr2=0x7f408415b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f407400c3f0 tx=0x7f407400c8b0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 shutdown_connections 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f40840c9630 0x7f40841481a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f406406f430 0x7f4064071940 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4084154780 0x7f4084150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 --2- 172.21.15.81:0/4209460216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4084072a60 0x7f408415b690 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 >> 172.21.15.81:0/4209460216 conn(0x7f4084068880 msgr2=0x7f4084068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 shutdown_connections 2024-05-27T06:57:49.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:49.210+0000 7f4083e00640 1 -- 172.21.15.81:0/4209460216 wait complete. 2024-05-27T06:57:49.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:51.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:51.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 Processor -- start 2024-05-27T06:57:51.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 -- start start 2024-05-27T06:57:51.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b00c5b60 0x7fa7b00c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7b0161cd0 0x7fa7b01640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b0072840 con 0x7fa7b0072430 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b00c5040 con 0x7fa7b0161cd0 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.390+0000 7fa7b8800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b00c75c0 con 0x7fa7b00c5b60 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7b0161cd0 0x7fa7b01640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7b0161cd0 0x7fa7b01640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33948/0 (socket says 172.21.15.81:33948) 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 -- 172.21.15.81:0/1341586026 learned_addr learned my addr 172.21.15.81:0/1341586026 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:51.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b5400640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b00c5b60 0x7fa7b00c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 -- 172.21.15.81:0/1341586026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b00c5b60 msgr2=0x7fa7b00c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:51.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b00c5b60 0x7fa7b00c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 -- 172.21.15.81:0/1341586026 >> v1:172.21.15.106:6789/0 conn(0x7fa7b0072430 legacy=0x7fa7b0159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 -- 172.21.15.81:0/1341586026 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b00c5270 con 0x7fa7b0161cd0 2024-05-27T06:57:51.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b6800640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0161cd0 0x7fa7b01640f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00d150 tx=0x7fa7ac01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734ed1bdaa1e57cc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:51.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b4a00640 1 -- 172.21.15.81:0/1341586026 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac00b2d0 con 0x7fa7b0161cd0 2024-05-27T06:57:51.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b4a00640 1 -- 172.21.15.81:0/1341586026 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7ac00b470 con 0x7fa7b0161cd0 2024-05-27T06:57:51.387 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.394+0000 7fa7b4a00640 1 -- 172.21.15.81:0/1341586026 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0a7a20 con 0x7fa7b0161cd0 2024-05-27T06:57:51.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- 172.21.15.81:0/1341586026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0161cd0 msgr2=0x7fa7b01640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0161cd0 0x7fa7b01640f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00d150 tx=0x7fa7ac01ec30 comp rx=0 tx=0).stop 2024-05-27T06:57:51.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- 172.21.15.81:0/1341586026 shutdown_connections 2024-05-27T06:57:51.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0161cd0 0x7fa7b01640f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1341586026 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b00c5b60 0x7fa7b00c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- 172.21.15.81:0/1341586026 >> 172.21.15.81:0/1341586026 conn(0x7fa7b0070700 msgr2=0x7fa7b00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:51.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- 172.21.15.81:0/1341586026 shutdown_connections 2024-05-27T06:57:51.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- 172.21.15.81:0/1341586026 wait complete. 2024-05-27T06:57:51.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 Processor -- start 2024-05-27T06:57:51.390 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.398+0000 7fa7b8800640 1 -- start start 2024-05-27T06:57:51.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7b00c5b60 0x7fa7b01553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b0161cd0 0x7fa7b01558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00ce6f0 con 0x7fa7b0072430 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00c7dd0 con 0x7fa7b00c5b60 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b00c5610 con 0x7fa7b0161cd0 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33954/0 (socket says 172.21.15.81:33954) 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 -- 172.21.15.81:0/1920821671 learned_addr learned my addr 172.21.15.81:0/1920821671 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:51.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5400640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7b00c5b60 0x7fa7b01553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b6800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b0161cd0 0x7fa7b01558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 -- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b0161cd0 msgr2=0x7fa7b01558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b0161cd0 0x7fa7b01558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 -- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7b00c5b60 msgr2=0x7fa7b01553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7b00c5b60 0x7fa7b01553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 -- 172.21.15.81:0/1920821671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b00cb7a0 con 0x7fa7b0072430 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b5e00640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c3c0 tx=0x7fa7a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4017070 con 0x7fa7b0072430 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a40052d0 con 0x7fa7b0072430 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4003840 con 0x7fa7b0072430 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b00cd5b0 con 0x7fa7b0072430 2024-05-27T06:57:51.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b01f6ce0 con 0x7fa7b0072430 2024-05-27T06:57:51.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.402+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b00cb970 con 0x7fa7b0072430 2024-05-27T06:57:51.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.406+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa7a401a070 con 0x7fa7b0072430 2024-05-27T06:57:51.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.406+0000 7fa7aa000640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 0x7fa794071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:51.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.406+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa7a408e6e0 con 0x7fa7b0072430 2024-05-27T06:57:51.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.406+0000 7fa7b5400640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 0x7fa794071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:51.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.406+0000 7fa7b5400640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 0x7fa794071910 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa7b006dc10 tx=0x7fa7a0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:51.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.410+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa7a4005480 con 0x7fa7b0072430 2024-05-27T06:57:51.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.598+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7b006a560 con 0x7fa7b0072430 2024-05-27T06:57:51.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7aa000640 1 -- 172.21.15.81:0/1920821671 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fa7a4001090 con 0x7fa7b0072430 2024-05-27T06:57:51.595 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:51.595 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:51.595 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 msgr2=0x7fa794071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 0x7fa794071910 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa7b006dc10 tx=0x7fa7a0009190 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 msgr2=0x7fa7b00cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400c3c0 tx=0x7fa7a400c880 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 shutdown_connections 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa7b0161cd0 0x7fa7b01558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa79406f400 0x7fa794071910 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa7b00c5b60 0x7fa7b01553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 --2- 172.21.15.81:0/1920821671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b0072430 0x7fa7b00cd070 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 >> 172.21.15.81:0/1920821671 conn(0x7fa7b0070700 msgr2=0x7fa7b00c7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 shutdown_connections 2024-05-27T06:57:51.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:51.602+0000 7fa7b8800640 1 -- 172.21.15.81:0/1920821671 wait complete. 2024-05-27T06:57:51.614 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:51.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~107s 2024-05-27T06:57:53.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:53.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.782+0000 7f7aaee00640 1 Processor -- start 2024-05-27T06:57:53.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.782+0000 7f7aaee00640 1 -- start start 2024-05-27T06:57:53.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aaee00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 0x7f7aa80c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:53.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aaee00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa815db00 con 0x7f7aa815a510 2024-05-27T06:57:53.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aaee00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aa8073860 con 0x7f7aa81546f0 2024-05-27T06:57:53.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aaee00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa815e6f0 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aace00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7aa815a510 0x7f7aa815a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33602/0 (socket says 172.21.15.81:33602) 2024-05-27T06:57:53.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aace00640 1 -- 172.21.15.81:0/1007342467 learned_addr learned my addr 172.21.15.81:0/1007342467 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7a9f400640 1 --2- 172.21.15.81:0/1007342467 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 0x7f7aa80c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1473861025 0 0) 0x7f7aa815db00 con 0x7f7aa815a510 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a8c003530 con 0x7f7aa815a510 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541757718 0 0) 0x7f7aa8073860 con 0x7f7aa81546f0 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa815db00 con 0x7f7aa81546f0 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7a9f400640 1 -- 172.21.15.81:0/1007342467 >> v1:172.21.15.106:6789/0 conn(0x7f7aa81546f0 legacy=0x7f7aa8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7a9f400640 1 -- 172.21.15.81:0/1007342467 >> v1:172.21.15.81:6789/0 conn(0x7f7aa815a510 legacy=0x7f7aa815a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.786+0000 7f7a9f400640 1 -- 172.21.15.81:0/1007342467 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa815ddc0 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7a9f400640 1 --2- 172.21.15.81:0/1007342467 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 0x7f7aa80c73e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a900097f0 tx=0x7f7a9001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae0f79103b97f62 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:53.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a900b5070 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a900a89d0 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aa7400640 1 -- 172.21.15.81:0/1007342467 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a900a8da0 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 -- 172.21.15.81:0/1007342467 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 msgr2=0x7f7aa80c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 --2- 172.21.15.81:0/1007342467 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 0x7f7aa80c73e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a900097f0 tx=0x7f7a9001ecd0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 -- 172.21.15.81:0/1007342467 shutdown_connections 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 --2- 172.21.15.81:0/1007342467 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa80c6fd0 0x7f7aa80c73e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 -- 172.21.15.81:0/1007342467 >> 172.21.15.81:0/1007342467 conn(0x7f7aa8070700 msgr2=0x7f7aa80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 -- 172.21.15.81:0/1007342467 shutdown_connections 2024-05-27T06:57:53.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.790+0000 7f7aaee00640 1 -- 172.21.15.81:0/1007342467 wait complete. 2024-05-27T06:57:53.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 Processor -- start 2024-05-27T06:57:53.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 -- start start 2024-05-27T06:57:53.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 0x7f7aa814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:53.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa815a510 0x7f7aa8155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa815e380 con 0x7f7aa81546f0 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa80755b0 con 0x7f7aa80c6fd0 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aaee00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7aa8154b00 con 0x7f7aa815a510 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33958/0 (socket says 172.21.15.81:33958) 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 -- 172.21.15.81:0/3779141973 learned_addr learned my addr 172.21.15.81:0/3779141973 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aa7e00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 0x7f7aa814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aace00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa815a510 0x7f7aa8155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 -- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa815a510 msgr2=0x7f7aa8155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa815a510 0x7f7aa8155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 -- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 msgr2=0x7f7aa814f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 0x7f7aa814f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7a9f400640 1 -- 172.21.15.81:0/3779141973 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa81f6e40 con 0x7f7aa81546f0 2024-05-27T06:57:53.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.794+0000 7f7aa7e00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 0x7f7aa814f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:53.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7a9f400640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7a900b3040 tx=0x7f7a9001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:53.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a900b5040 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a90004220 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa81f2e10 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a90003620 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa8156900 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa81f2c00 con 0x7f7aa81546f0 2024-05-27T06:57:53.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f7a900c2060 con 0x7f7aa81546f0 2024-05-27T06:57:53.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 0x7f7a84071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:53.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f7a9012bd00 con 0x7f7aa81546f0 2024-05-27T06:57:53.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.798+0000 7f7aa7e00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 0x7f7a84071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:53.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.802+0000 7f7aa7e00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 0x7f7a84071910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7a98001130 tx=0x7f7a98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:53.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.806+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f7a90018830 con 0x7f7aa81546f0 2024-05-27T06:57:53.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.994+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7aa806a560 con 0x7f7aa81546f0 2024-05-27T06:57:53.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aa4c00640 1 -- 172.21.15.81:0/3779141973 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f7a900f8850 con 0x7f7aa81546f0 2024-05-27T06:57:53.988 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:53.988 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:53.988 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 msgr2=0x7f7a84071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 0x7f7a84071910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7a98001130 tx=0x7f7a98009040 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 msgr2=0x7f7aa8154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7a900b3040 tx=0x7f7a9001ba70 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 shutdown_connections 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f7aa815a510 0x7f7aa8155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f7a8406f400 0x7f7a84071910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aa81546f0 0x7f7aa8154e00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 --2- 172.21.15.81:0/3779141973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7aa80c6fd0 0x7f7aa814f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 >> 172.21.15.81:0/3779141973 conn(0x7f7aa8070700 msgr2=0x7f7aa81515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 shutdown_connections 2024-05-27T06:57:53.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:53.998+0000 7f7aaee00640 1 -- 172.21.15.81:0/3779141973 wait complete. 2024-05-27T06:57:54.010 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:57:56.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:56.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.178+0000 7fd80de00640 1 Processor -- start 2024-05-27T06:57:56.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.178+0000 7fd80de00640 1 -- start start 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd80de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd8080c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd80de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd808072840 con 0x7fd808072430 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd80de00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8080c5270 con 0x7fd808161cd0 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd80de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd8080c5040 con 0x7fd8080c5b60 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd808072430 0x7fd808159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38166/0 (socket says 172.21.15.81:38166) 2024-05-27T06:57:56.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 -- 172.21.15.81:0/3907336757 learned_addr learned my addr 172.21.15.81:0/3907336757 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806000640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd8080c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097404236 0 0) 0x7fd808072840 con 0x7fd808072430 2024-05-27T06:57:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7ec003530 con 0x7fd808072430 2024-05-27T06:57:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3300739758 0 0) 0x7fd7ec003530 con 0x7fd808072430 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 msgr2=0x7fd8080c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd8080c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 >> v1:172.21.15.106:6789/0 conn(0x7fd808161cd0 legacy=0x7fd8081640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8080c7810 con 0x7fd808072430 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fd7f8001a80 con 0x7fd808072430 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 >> v1:172.21.15.81:6789/0 conn(0x7fd808072430 legacy=0x7fd808159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd7ec003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd7ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7ec004d50 0x7fd7ec005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec00bb60 con 0x7fd7ec003a40 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ec00bd10 con 0x7fd7ec004d50 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ec00e120 con 0x7fd8080c5b60 2024-05-27T06:57:56.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd7ec003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806000640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd7ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd807400640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7ec004d50 0x7fd7ec005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 -- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 msgr2=0x7fd7ec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080c5b60 0x7fd7ec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 -- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7ec004d50 msgr2=0x7fd7ec005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7ec004d50 0x7fd7ec005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.182+0000 7fd806a00640 1 -- 172.21.15.81:0/3907336757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ec003530 con 0x7fd7ec003a40 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd806a00640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd7ec003eb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800a890 tx=0x7fd7f801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829a858252fe6f16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800a4e0 con 0x7fd7ec003a40 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f80ad630 con 0x7fd7ec003a40 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd805600640 1 -- 172.21.15.81:0/3907336757 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80ada20 con 0x7fd7ec003a40 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 -- 172.21.15.81:0/3907336757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 msgr2=0x7fd7ec003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 --2- 172.21.15.81:0/3907336757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd7ec003eb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800a890 tx=0x7fd7f801d2a0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.181 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd807400640 1 -- 172.21.15.81:0/3907336757 reap_dead start 2024-05-27T06:57:56.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 -- 172.21.15.81:0/3907336757 shutdown_connections 2024-05-27T06:57:56.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 -- 172.21.15.81:0/3907336757 >> 172.21.15.81:0/3907336757 conn(0x7fd808070700 msgr2=0x7fd8080cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:56.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 -- 172.21.15.81:0/3907336757 shutdown_connections 2024-05-27T06:57:56.182 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.186+0000 7fd80de00640 1 -- 172.21.15.81:0/3907336757 wait complete. 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 Processor -- start 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 -- start start 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 0x7fd808151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8080c3520 0x7fd8080c3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080c7dd0 con 0x7fd7ec003a40 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd8080c75c0 con 0x7fd8080c3520 2024-05-27T06:57:56.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd80de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd8080ce6f0 con 0x7fd8080cd0d0 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33578/0 (socket says 172.21.15.81:33578) 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 -- 172.21.15.81:0/1448275422 learned_addr learned my addr 172.21.15.81:0/1448275422 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd807400640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8080c3520 0x7fd8080c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806000640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 0x7fd808151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 -- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 msgr2=0x7fd808151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 0x7fd808151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 -- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8080c3520 msgr2=0x7fd8080c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8080c3520 0x7fd8080c3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 -- 172.21.15.81:0/1448275422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081465c0 con 0x7fd7ec003a40 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806000640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 0x7fd808151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.190+0000 7fd806a00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801b9a0 tx=0x7fd7ec005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80b2040 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f8001840 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8081468a0 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80ad7e0 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd808147a40 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80806a560 con 0x7fd7ec003a40 2024-05-27T06:57:56.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7f80ad980 con 0x7fd7ec003a40 2024-05-27T06:57:56.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 0x7fd7e8071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:56.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd7f81290c0 con 0x7fd7ec003a40 2024-05-27T06:57:56.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.194+0000 7fd806000640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 0x7fd7e8071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:56.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.198+0000 7fd806000640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 0x7fd7e8071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd808150570 tx=0x7fd7f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:56.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.202+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd7f80bd030 con 0x7fd7ec003a40 2024-05-27T06:57:56.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.390+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8080c5940 con 0x7fd7ec003a40 2024-05-27T06:57:56.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.390+0000 7fd7ff400640 1 -- 172.21.15.81:0/1448275422 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd7f8017050 con 0x7fd7ec003a40 2024-05-27T06:57:56.382 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:56.382 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:56.383 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 msgr2=0x7fd7e8071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 0x7fd7e8071940 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd808150570 tx=0x7fd7f4008040 comp rx=0 tx=0).stop 2024-05-27T06:57:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 msgr2=0x7fd8080cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd7f801b9a0 tx=0x7fd7ec005840 comp rx=0 tx=0).stop 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 shutdown_connections 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd8080cd0d0 0x7fd808151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7e806f430 0x7fd7e8071940 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8080c3520 0x7fd8080c3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 --2- 172.21.15.81:0/1448275422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7ec003a40 0x7fd8080cc0d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 >> 172.21.15.81:0/1448275422 conn(0x7fd808070700 msgr2=0x7fd808153030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 shutdown_connections 2024-05-27T06:57:56.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:56.394+0000 7fd80de00640 1 -- 172.21.15.81:0/1448275422 wait complete. 2024-05-27T06:57:56.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:57:58.405 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:57:58.459 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~114s 2024-05-27T06:57:58.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 Processor -- start 2024-05-27T06:57:58.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 -- start start 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fac040cfdb0 0x7fac040d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fac04154200 0x7fac04154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac040c78b0 con 0x7fac040ca770 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fac04154b50 con 0x7fac04154200 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0c600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fac040c6ee0 con 0x7fac040cfdb0 2024-05-27T06:57:58.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fac04154200 0x7fac04154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac09c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fac040ca770 0x7fac040cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38172/0 (socket says 172.21.15.81:38172) 2024-05-27T06:57:58.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac09c00640 1 -- 172.21.15.81:0/144302709 learned_addr learned my addr 172.21.15.81:0/144302709 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:57:58.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac09200640 1 --2- 172.21.15.81:0/144302709 >> v2:172.21.15.106:3300/0 conn(0x7fac040cfdb0 0x7fac040d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0a600640 1 -- 172.21.15.81:0/144302709 >> v2:172.21.15.106:3300/0 conn(0x7fac040cfdb0 msgr2=0x7fac040d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:57:58.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0a600640 1 --2- 172.21.15.81:0/144302709 >> v2:172.21.15.106:3300/0 conn(0x7fac040cfdb0 0x7fac040d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0a600640 1 -- 172.21.15.81:0/144302709 >> v1:172.21.15.81:6789/0 conn(0x7fac040ca770 legacy=0x7fac040cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac0a600640 1 -- 172.21.15.81:0/144302709 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac040c71c0 con 0x7fac04154200 2024-05-27T06:57:58.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.574+0000 7fac03e00640 1 -- 172.21.15.81:0/144302709 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770136075 0 0) 0x7fac040c78b0 con 0x7fac040ca770 2024-05-27T06:57:58.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0a600640 1 --2- 172.21.15.81:0/144302709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac04154610 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fac04073180 tx=0x7fabfc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e6db2ea2e7f342 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac03e00640 1 -- 172.21.15.81:0/144302709 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b1070 con 0x7fac04154200 2024-05-27T06:57:58.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac03e00640 1 -- 172.21.15.81:0/144302709 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabfc0ac5f0 con 0x7fac04154200 2024-05-27T06:57:58.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac03e00640 1 -- 172.21.15.81:0/144302709 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0ac9c0 con 0x7fac04154200 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 -- 172.21.15.81:0/144302709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 msgr2=0x7fac04154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 --2- 172.21.15.81:0/144302709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac04154610 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fac04073180 tx=0x7fabfc01c480 comp rx=0 tx=0).stop 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 -- 172.21.15.81:0/144302709 shutdown_connections 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 --2- 172.21.15.81:0/144302709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac04154610 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 --2- 172.21.15.81:0/144302709 >> v2:172.21.15.106:3300/0 conn(0x7fac040cfdb0 0x7fac040d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 -- 172.21.15.81:0/144302709 >> 172.21.15.81:0/144302709 conn(0x7fac04068ae0 msgr2=0x7fac04068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:58.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 -- 172.21.15.81:0/144302709 shutdown_connections 2024-05-27T06:57:58.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.578+0000 7fac0c600640 1 -- 172.21.15.81:0/144302709 wait complete. 2024-05-27T06:57:58.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 Processor -- start 2024-05-27T06:57:58.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 -- start start 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac040ca770 0x7fac040cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac040cfdb0 0x7fac041f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac040cc050 con 0x7fac04154200 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fac040cbed0 con 0x7fac040cfdb0 2024-05-27T06:57:58.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0c600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fac04167590 con 0x7fac040ca770 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33596/0 (socket says 172.21.15.81:33596) 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 -- 172.21.15.81:0/357085400 learned_addr learned my addr 172.21.15.81:0/357085400 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac09200640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac040cfdb0 0x7fac041f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac09c00640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac040ca770 0x7fac040cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 -- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac040ca770 msgr2=0x7fac040cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac040ca770 0x7fac040cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 -- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac040cfdb0 msgr2=0x7fac041f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac040cfdb0 0x7fac041f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 -- 172.21.15.81:0/357085400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac041f5dc0 con 0x7fac04154200 2024-05-27T06:57:58.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.582+0000 7fac0a600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fabfc01ab30 tx=0x7fabfc019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0bc070 con 0x7fac04154200 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabfc004030 con 0x7fac04154200 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b1040 con 0x7fac04154200 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac04157de0 con 0x7fac04154200 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac041563a0 con 0x7fac04154200 2024-05-27T06:57:58.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac0406a560 con 0x7fac04154200 2024-05-27T06:57:58.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fabfc003620 con 0x7fac04154200 2024-05-27T06:57:58.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 0x7fabe4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:57:58.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.586+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fabfc1283b0 con 0x7fac04154200 2024-05-27T06:57:58.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.590+0000 7fac09c00640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 0x7fabe4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:57:58.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.590+0000 7fac09c00640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 0x7fabe4071940 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fabf40012f0 tx=0x7fabf4001690 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:57:58.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.594+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fabfc0a9070 con 0x7fac04154200 2024-05-27T06:57:58.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.786+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac041f51c0 con 0x7fac04154200 2024-05-27T06:57:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.786+0000 7fac01600640 1 -- 172.21.15.81:0/357085400 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fabfc016070 con 0x7fac04154200 2024-05-27T06:57:58.779 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:57:58.779 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:57:58.779 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:57:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 msgr2=0x7fabe4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 0x7fabe4071940 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fabf40012f0 tx=0x7fabf4001690 comp rx=0 tx=0).stop 2024-05-27T06:57:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 msgr2=0x7fac0414f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fabfc01ab30 tx=0x7fabfc019220 comp rx=0 tx=0).stop 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 shutdown_connections 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fabe406f430 0x7fabe4071940 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac04154200 0x7fac0414f570 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fac040cfdb0 0x7fac041f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 --2- 172.21.15.81:0/357085400 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fac040ca770 0x7fac040cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 >> 172.21.15.81:0/357085400 conn(0x7fac04068ae0 msgr2=0x7fac041645e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 shutdown_connections 2024-05-27T06:57:58.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:57:58.790+0000 7fac0c600640 1 -- 172.21.15.81:0/357085400 wait complete. 2024-05-27T06:57:58.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:00.802 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:00.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.970+0000 7f56e5a00640 1 Processor -- start 2024-05-27T06:58:00.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.970+0000 7f56e5a00640 1 -- start start 2024-05-27T06:58:00.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56e5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e00c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56e5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56e0161cd0 0x7f56e01640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56e5a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e0072840 con 0x7f56e0072430 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56e5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f56e00c5040 con 0x7f56e0161cd0 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56e5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f56e00c75c0 con 0x7f56e00c5b60 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56e0161cd0 0x7f56e01640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56e0161cd0 0x7f56e01640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33606/0 (socket says 172.21.15.81:33606) 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 -- 172.21.15.81:0/850867169 learned_addr learned my addr 172.21.15.81:0/850867169 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56d6000640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e00c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 -- 172.21.15.81:0/850867169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 msgr2=0x7f56e00c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e00c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 -- 172.21.15.81:0/850867169 >> v1:172.21.15.106:6789/0 conn(0x7f56e0072430 legacy=0x7f56e0159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 -- 172.21.15.81:0/850867169 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e00c5270 con 0x7f56e0161cd0 2024-05-27T06:58:00.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.974+0000 7f56df400640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0161cd0 0x7f56e01640f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009e50 tx=0x7f56d801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32988642accddcd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56de000640 1 -- 172.21.15.81:0/850867169 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80a5700 con 0x7f56e0161cd0 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56de000640 1 -- 172.21.15.81:0/850867169 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d80a58a0 con 0x7f56e0161cd0 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56de000640 1 -- 172.21.15.81:0/850867169 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d80a5c70 con 0x7f56e0161cd0 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- 172.21.15.81:0/850867169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0161cd0 msgr2=0x7f56e01640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0161cd0 0x7f56e01640f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009e50 tx=0x7f56d801ca30 comp rx=0 tx=0).stop 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- 172.21.15.81:0/850867169 shutdown_connections 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0161cd0 0x7f56e01640f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 --2- 172.21.15.81:0/850867169 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e00c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- 172.21.15.81:0/850867169 >> 172.21.15.81:0/850867169 conn(0x7f56e0070700 msgr2=0x7f56e00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- 172.21.15.81:0/850867169 shutdown_connections 2024-05-27T06:58:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- 172.21.15.81:0/850867169 wait complete. 2024-05-27T06:58:00.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 Processor -- start 2024-05-27T06:58:00.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.978+0000 7f56e5a00640 1 -- start start 2024-05-27T06:58:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e0163660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e0161cd0 0x7f56e01542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56e00ce6f0 con 0x7f56e0072430 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f56e00c7dd0 con 0x7f56e0161cd0 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f56e00c5610 con 0x7f56e00c5b60 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33608/0 (socket says 172.21.15.81:33608) 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 -- 172.21.15.81:0/3482822194 learned_addr learned my addr 172.21.15.81:0/3482822194 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56df400640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e0161cd0 0x7f56e01542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56d6000640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e0163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 -- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 msgr2=0x7f56e0163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e0163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 -- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e0161cd0 msgr2=0x7f56e01542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e0161cd0 0x7f56e01542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 -- 172.21.15.81:0/3482822194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e0159270 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56dea00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f56d000c430 tx=0x7f56d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0017070 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d00052d0 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e0157350 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d00054c0 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e01575f0 con 0x7f56e0072430 2024-05-27T06:58:00.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.982+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e006a560 con 0x7f56e0072430 2024-05-27T06:58:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.986+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f56d001a070 con 0x7f56e0072430 2024-05-27T06:58:00.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.986+0000 7f56d7400640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 0x7f56bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:00.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.986+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f56d008e7d0 con 0x7f56e0072430 2024-05-27T06:58:00.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.986+0000 7f56d6000640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 0x7f56bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:00.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.986+0000 7f56d6000640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 0x7f56bc071910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56e00c9120 tx=0x7f56c8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:00.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:00.990+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f56e0157350 con 0x7f56e0072430 2024-05-27T06:58:01.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56e0075e70 con 0x7f56e0072430 2024-05-27T06:58:01.172 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56d7400640 1 -- 172.21.15.81:0/3482822194 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f56d005ae70 con 0x7f56e0072430 2024-05-27T06:58:01.173 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:01.173 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:01.173 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 msgr2=0x7f56bc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 0x7f56bc071910 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56e00c9120 tx=0x7f56c8009190 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 msgr2=0x7f56e0158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f56d000c430 tx=0x7f56d000c8f0 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 shutdown_connections 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f56e00c5b60 0x7f56e0163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f56bc06f400 0x7f56bc071910 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56e0161cd0 0x7f56e01542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 --2- 172.21.15.81:0/3482822194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e0072430 0x7f56e0158ba0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:01.175 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 >> 172.21.15.81:0/3482822194 conn(0x7f56e0070700 msgr2=0x7f56e00c65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:01.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 shutdown_connections 2024-05-27T06:58:01.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:01.182+0000 7f56e5a00640 1 -- 172.21.15.81:0/3482822194 wait complete. 2024-05-27T06:58:01.195 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:03.196 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 Processor -- start 2024-05-27T06:58:03.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 -- start start 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 0x7f9c08159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9c08161cd0 0x7f9c081640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c08072840 con 0x7f9c0815a040 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f9c080c5040 con 0x7f9c08161cd0 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c0da00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f9c080c75c0 con 0x7f9c08072430 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9bfe000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9c0815a040 0x7f9c080c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38174/0 (socket says 172.21.15.81:38174) 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9bfe000640 1 -- 172.21.15.81:0/302575764 learned_addr learned my addr 172.21.15.81:0/302575764 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c07400640 1 --2- 172.21.15.81:0/302575764 >> v2:172.21.15.81:3300/0 conn(0x7f9c08161cd0 0x7f9c081640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c06a00640 1 --2- 172.21.15.81:0/302575764 >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 0x7f9c08159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c06000640 1 -- 172.21.15.81:0/302575764 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625448935 0 0) 0x7f9c08072840 con 0x7f9c0815a040 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c06000640 1 -- 172.21.15.81:0/302575764 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bec003530 con 0x7f9c0815a040 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c07400640 1 -- 172.21.15.81:0/302575764 >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 msgr2=0x7f9c08159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c07400640 1 --2- 172.21.15.81:0/302575764 >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 0x7f9c08159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c07400640 1 -- 172.21.15.81:0/302575764 >> v1:172.21.15.81:6789/0 conn(0x7f9c0815a040 legacy=0x7f9c080c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c07400640 1 -- 172.21.15.81:0/302575764 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c080c5270 con 0x7f9c08161cd0 2024-05-27T06:58:03.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.366+0000 7f9c06a00640 1 --2- 172.21.15.81:0/302575764 >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 0x7f9c08159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:03.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c07400640 1 --2- 172.21.15.81:0/302575764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c081640f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9c08071c00 tx=0x7f9c0001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929921c43c8db238 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:03.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c06000640 1 -- 172.21.15.81:0/302575764 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000b2070 con 0x7f9c08161cd0 2024-05-27T06:58:03.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c06000640 1 -- 172.21.15.81:0/302575764 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000a57f0 con 0x7f9c08161cd0 2024-05-27T06:58:03.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c06000640 1 -- 172.21.15.81:0/302575764 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000a5bc0 con 0x7f9c08161cd0 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 -- 172.21.15.81:0/302575764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 msgr2=0x7f9c081640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 --2- 172.21.15.81:0/302575764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c081640f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f9c08071c00 tx=0x7f9c0001c930 comp rx=0 tx=0).stop 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 -- 172.21.15.81:0/302575764 shutdown_connections 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 --2- 172.21.15.81:0/302575764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c081640f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 --2- 172.21.15.81:0/302575764 >> v2:172.21.15.106:3300/0 conn(0x7f9c08072430 0x7f9c08159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 -- 172.21.15.81:0/302575764 >> 172.21.15.81:0/302575764 conn(0x7f9c08070700 msgr2=0x7f9c080cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 -- 172.21.15.81:0/302575764 shutdown_connections 2024-05-27T06:58:03.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.370+0000 7f9c0da00640 1 -- 172.21.15.81:0/302575764 wait complete. 2024-05-27T06:58:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 Processor -- start 2024-05-27T06:58:03.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 -- start start 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 0x7f9c080ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 0x7f9c08142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c080ce6f0 con 0x7f9c08161cd0 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9c080c7dd0 con 0x7f9c0815a040 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c0da00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9c080c5610 con 0x7f9c08072430 2024-05-27T06:58:03.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33624/0 (socket says 172.21.15.81:33624) 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 -- 172.21.15.81:0/2464924628 learned_addr learned my addr 172.21.15.81:0/2464924628 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9bfe000640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 0x7f9c08142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c06a00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 0x7f9c080ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 -- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 msgr2=0x7f9c080ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 0x7f9c080ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 -- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 msgr2=0x7f9c08142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 0x7f9c08142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c07400640 1 -- 172.21.15.81:0/2464924628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c08162900 con 0x7f9c08161cd0 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9bfe000640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 0x7f9c08142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.374+0000 7f9c06a00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 0x7f9c080ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9c07400640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9c00019540 tx=0x7f9c000196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:03.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000b2050 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c000be070 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c000ad890 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c081f9740 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c081f6390 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c0806a560 con 0x7f9c08161cd0 2024-05-27T06:58:03.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f9c000ada30 con 0x7f9c08161cd0 2024-05-27T06:58:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 0x7f9be4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f9c00129f90 con 0x7f9c08161cd0 2024-05-27T06:58:03.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.378+0000 7f9c06a00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 0x7f9be4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:03.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.382+0000 7f9c06a00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 0x7f9be4071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c08070fe0 tx=0x7f9bf8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:03.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.382+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9c000c7020 con 0x7f9c08161cd0 2024-05-27T06:58:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c081f65c0 con 0x7f9c08161cd0 2024-05-27T06:58:03.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9bff400640 1 -- 172.21.15.81:0/2464924628 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f9c00016520 con 0x7f9c08161cd0 2024-05-27T06:58:03.565 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:03.565 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:03.565 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 msgr2=0x7f9be4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 0x7f9be4071940 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c08070fe0 tx=0x7f9bf8008040 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 msgr2=0x7f9c08143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9c00019540 tx=0x7f9c000196d0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 shutdown_connections 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f9be406f430 0x7f9be4071940 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c08161cd0 0x7f9c08143470 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c0815a040 0x7f9c08142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 --2- 172.21.15.81:0/2464924628 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f9c08072430 0x7f9c080ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:03.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 >> 172.21.15.81:0/2464924628 conn(0x7f9c08070700 msgr2=0x7f9c080c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:03.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 shutdown_connections 2024-05-27T06:58:03.568 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:03.574+0000 7f9c0da00640 1 -- 172.21.15.81:0/2464924628 wait complete. 2024-05-27T06:58:03.587 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:04.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:58:04.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:58:05.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~121s 2024-05-27T06:58:05.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:05.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 Processor -- start 2024-05-27T06:58:05.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 -- start start 2024-05-27T06:58:05.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4448156bc0 0x7f44480cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44480c99f0 con 0x7f44481603b0 2024-05-27T06:58:05.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4448156fd0 con 0x7f4448156bc0 2024-05-27T06:58:05.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444fe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f44481607c0 con 0x7f44480cc830 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4448156bc0 0x7f44480cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444de00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f44481603b0 0x7f4448071f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:51890/0 (socket says 172.21.15.81:51890) 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:35144/0 (socket says 172.21.15.81:35144) 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4448156bc0 0x7f44480cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:38124/0 (socket says 172.21.15.81:38124) 2024-05-27T06:58:05.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444de00640 1 -- 172.21.15.81:0/2703942209 learned_addr learned my addr 172.21.15.81:0/2703942209 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444d400640 1 -- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 msgr2=0x7f444815fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444d400640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.762+0000 7f444d400640 1 -- 172.21.15.81:0/2703942209 >> v1:172.21.15.106:6789/0 conn(0x7f44481603b0 legacy=0x7f4448071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f444d400640 1 -- 172.21.15.81:0/2703942209 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444815c980 con 0x7f4448156bc0 2024-05-27T06:58:05.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f444ca00640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:58:05.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f444d400640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f44480cc2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097f0 tx=0x7f443c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b787c84f88e680d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:05.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f4447e00640 1 -- 172.21.15.81:0/2703942209 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c0b5070 con 0x7f4448156bc0 2024-05-27T06:58:05.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f4447e00640 1 -- 172.21.15.81:0/2703942209 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f443c0a8990 con 0x7f4448156bc0 2024-05-27T06:58:05.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.766+0000 7f4447e00640 1 -- 172.21.15.81:0/2703942209 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c0a8d60 con 0x7f4448156bc0 2024-05-27T06:58:05.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 msgr2=0x7f44480cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f44480cc2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097f0 tx=0x7f443c01ec90 comp rx=0 tx=0).stop 2024-05-27T06:58:05.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- 172.21.15.81:0/2703942209 shutdown_connections 2024-05-27T06:58:05.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f444815fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 --2- 172.21.15.81:0/2703942209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f44480cc2f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- 172.21.15.81:0/2703942209 >> 172.21.15.81:0/2703942209 conn(0x7f4448068cb0 msgr2=0x7f44480745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:05.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- 172.21.15.81:0/2703942209 shutdown_connections 2024-05-27T06:58:05.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- 172.21.15.81:0/2703942209 wait complete. 2024-05-27T06:58:05.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 Processor -- start 2024-05-27T06:58:05.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.770+0000 7f444fe00640 1 -- start start 2024-05-27T06:58:05.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f44481e7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f4448155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f444815d220 con 0x7f44481603b0 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f444815cf40 con 0x7f4448156bc0 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f444815d500 con 0x7f44480cc830 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45830/0 (socket says 172.21.15.81:45830) 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 -- 172.21.15.81:0/1490600704 learned_addr learned my addr 172.21.15.81:0/1490600704 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444ca00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f4448155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444d400640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f44481e7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 -- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 msgr2=0x7f44481e7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f44481e7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 -- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 msgr2=0x7f4448155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f4448155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 -- 172.21.15.81:0/1490600704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44481e76a0 con 0x7f44481603b0 2024-05-27T06:58:05.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444de00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f444000ca80 tx=0x7f444000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:05.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440017070 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444000abd0 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440012400 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44481e63a0 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44481e6960 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444806a560 con 0x7f44481603b0 2024-05-27T06:58:05.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.774+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f444001a070 con 0x7f44481603b0 2024-05-27T06:58:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.778+0000 7f4445600640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 0x7f4428071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.778+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f444008f170 con 0x7f44481603b0 2024-05-27T06:58:05.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.778+0000 7f444d400640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 0x7f4428071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:05.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.778+0000 7f444d400640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 0x7f4428071910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f443c0b2000 tx=0x7f443c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:05.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.782+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f444005b780 con 0x7f44481603b0 2024-05-27T06:58:05.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.970+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44481e8670 con 0x7f44481603b0 2024-05-27T06:58:05.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f4445600640 1 -- 172.21.15.81:0/1490600704 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f444005fa80 con 0x7f44481603b0 2024-05-27T06:58:05.964 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:05.964 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:05.964 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 msgr2=0x7f4428071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 0x7f4428071910 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f443c0b2000 tx=0x7f443c008040 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 msgr2=0x7f44481545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f444000ca80 tx=0x7f444000cf40 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 shutdown_connections 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f442806f400 0x7f4428071910 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44481603b0 0x7f44481545b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4448156bc0 0x7f4448155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 --2- 172.21.15.81:0/1490600704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f44480cc830 0x7f44481e7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:05.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 >> 172.21.15.81:0/1490600704 conn(0x7f4448068cb0 msgr2=0x7f4448150520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:05.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 shutdown_connections 2024-05-27T06:58:05.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:05.974+0000 7f444fe00640 1 -- 172.21.15.81:0/1490600704 wait complete. 2024-05-27T06:58:05.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:58:07.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:08.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.154+0000 7f51e9a00640 1 Processor -- start 2024-05-27T06:58:08.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.154+0000 7f51e9a00640 1 -- start start 2024-05-27T06:58:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e9a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f51e4072430 0x7f51e4159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 0x7f51e40c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e9a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51e4072840 con 0x7f51e40c60a0 2024-05-27T06:58:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e9a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f51e40c5040 con 0x7f51e4072430 2024-05-27T06:58:08.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e9a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f51e40c75c0 con 0x7f51e415a040 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f51e4072430 0x7f51e4159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e3400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f51e40c60a0 0x7f51e415c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38654/0 (socket says 172.21.15.81:38654) 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e3400640 1 -- 172.21.15.81:0/4025610634 learned_addr learned my addr 172.21.15.81:0/4025610634 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51da000640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 0x7f51e40c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e2a00640 1 -- 172.21.15.81:0/4025610634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 msgr2=0x7f51e40c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e2a00640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 0x7f51e40c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e2a00640 1 -- 172.21.15.81:0/4025610634 >> v1:172.21.15.81:6789/0 conn(0x7f51e40c60a0 legacy=0x7f51e415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51e2a00640 1 -- 172.21.15.81:0/4025610634 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e40c5270 con 0x7f51e4072430 2024-05-27T06:58:08.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.158+0000 7f51da000640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 0x7f51e40c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:08.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e2a00640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e4072430 0x7f51e4159b00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097c0 tx=0x7f51d401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c34fc802488d85c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e2000640 1 -- 172.21.15.81:0/4025610634 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d40b0070 con 0x7f51e4072430 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e2000640 1 -- 172.21.15.81:0/4025610634 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d40a4e50 con 0x7f51e4072430 2024-05-27T06:58:08.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e2000640 1 -- 172.21.15.81:0/4025610634 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d40ab820 con 0x7f51e4072430 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 -- 172.21.15.81:0/4025610634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e4072430 msgr2=0x7f51e4159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e4072430 0x7f51e4159b00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097c0 tx=0x7f51d401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 -- 172.21.15.81:0/4025610634 shutdown_connections 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e415a040 0x7f51e40c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 --2- 172.21.15.81:0/4025610634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e4072430 0x7f51e4159b00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 -- 172.21.15.81:0/4025610634 >> 172.21.15.81:0/4025610634 conn(0x7f51e4070700 msgr2=0x7f51e40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 -- 172.21.15.81:0/4025610634 shutdown_connections 2024-05-27T06:58:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.162+0000 7f51e9a00640 1 -- 172.21.15.81:0/4025610634 wait complete. 2024-05-27T06:58:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 Processor -- start 2024-05-27T06:58:08.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- start start 2024-05-27T06:58:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e4072430 0x7f51e41f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 0x7f51e40c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f51e40ce6f0 con 0x7f51e415a040 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f51e40c7dd0 con 0x7f51e4072430 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f51e40c5610 con 0x7f51e40c60a0 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45834/0 (socket says 172.21.15.81:45834) 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 -- 172.21.15.81:0/242452397 learned_addr learned my addr 172.21.15.81:0/242452397 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51da000640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 0x7f51e40c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e2a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e4072430 0x7f51e41f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 -- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 msgr2=0x7f51e40c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 0x7f51e40c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 -- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e4072430 msgr2=0x7f51e41f70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e4072430 0x7f51e41f70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 -- 172.21.15.81:0/242452397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e41f75f0 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51da000640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 0x7f51e40c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e3400640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00ca80 tx=0x7f51dc00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc017070 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51dc00ac30 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc012400 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e41409a0 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.166+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e414d0c0 con 0x7f51e415a040 2024-05-27T06:58:08.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e406a560 con 0x7f51e415a040 2024-05-27T06:58:08.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f51dc01a070 con 0x7f51e415a040 2024-05-27T06:58:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51db400640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 0x7f51c0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f51dc08f0d0 con 0x7f51e415a040 2024-05-27T06:58:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51e2a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 0x7f51c0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:08.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.170+0000 7f51e2a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 0x7f51c0071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51d40ad000 tx=0x7f51d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.174+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f51dc05b770 con 0x7f51e415a040 2024-05-27T06:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.366+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51e40c5940 con 0x7f51e415a040 2024-05-27T06:58:08.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.366+0000 7f51db400640 1 -- 172.21.15.81:0/242452397 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f51dc05fa70 con 0x7f51e415a040 2024-05-27T06:58:08.359 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:08.359 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:08.360 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:08.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 msgr2=0x7f51c0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 0x7f51c0071910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51d40ad000 tx=0x7f51d4008040 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 msgr2=0x7f51e414be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00ca80 tx=0x7f51dc00cf40 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 shutdown_connections 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f51e40c60a0 0x7f51e40c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f51c006f400 0x7f51c0071910 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f51e415a040 0x7f51e414be20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 --2- 172.21.15.81:0/242452397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f51e4072430 0x7f51e41f70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 >> 172.21.15.81:0/242452397 conn(0x7f51e4070700 msgr2=0x7f51e40ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 shutdown_connections 2024-05-27T06:58:08.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:08.370+0000 7f51e9a00640 1 -- 172.21.15.81:0/242452397 wait complete. 2024-05-27T06:58:08.382 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:10.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 Processor -- start 2024-05-27T06:58:10.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 -- start start 2024-05-27T06:58:10.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc0c9630 0x7f0dfc154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0dfc15dac0 0x7f0dfc15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0dfc072520 0x7f0dfc072930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc0c9a40 con 0x7f0dfc15dac0 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc15b0c0 con 0x7f0dfc072520 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e02600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc0c9c70 con 0x7f0dfc0c9630 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0dfc15dac0 0x7f0dfc15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0dfc15dac0 0x7f0dfc15ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45846/0 (socket says 172.21.15.81:45846) 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 -- 172.21.15.81:0/719049113 learned_addr learned my addr 172.21.15.81:0/719049113 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e01200640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc0c9630 0x7f0dfc154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0e01c00640 1 --2- 172.21.15.81:0/719049113 >> v2:172.21.15.106:3300/0 conn(0x7f0dfc072520 0x7f0dfc072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 -- 172.21.15.81:0/719049113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc0c9630 msgr2=0x7f0dfc154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:10.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc0c9630 0x7f0dfc154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 -- 172.21.15.81:0/719049113 >> v2:172.21.15.106:3300/0 conn(0x7f0dfc072520 msgr2=0x7f0dfc072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:10.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 --2- 172.21.15.81:0/719049113 >> v2:172.21.15.106:3300/0 conn(0x7f0dfc072520 0x7f0dfc072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.550+0000 7f0df3e00640 1 -- 172.21.15.81:0/719049113 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc15a4d0 con 0x7f0dfc15dac0 2024-05-27T06:58:10.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0df3e00640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc15ded0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0c8db0 tx=0x7f0de401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464c1e0bc561e087 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:10.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0dfbe00640 1 -- 172.21.15.81:0/719049113 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40b2070 con 0x7f0dfc15dac0 2024-05-27T06:58:10.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0dfbe00640 1 -- 172.21.15.81:0/719049113 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de40a9070 con 0x7f0dfc15dac0 2024-05-27T06:58:10.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0dfbe00640 1 -- 172.21.15.81:0/719049113 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40ad880 con 0x7f0dfc15dac0 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 -- 172.21.15.81:0/719049113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc15dac0 msgr2=0x7f0dfc15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc15ded0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0c8db0 tx=0x7f0de401c490 comp rx=0 tx=0).stop 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 -- 172.21.15.81:0/719049113 shutdown_connections 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 --2- 172.21.15.81:0/719049113 >> v2:172.21.15.106:3300/0 conn(0x7f0dfc072520 0x7f0dfc072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc15ded0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 --2- 172.21.15.81:0/719049113 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc0c9630 0x7f0dfc154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 -- 172.21.15.81:0/719049113 >> 172.21.15.81:0/719049113 conn(0x7f0dfc068880 msgr2=0x7f0dfc068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:10.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 -- 172.21.15.81:0/719049113 shutdown_connections 2024-05-27T06:58:10.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.554+0000 7f0e02600640 1 -- 172.21.15.81:0/719049113 wait complete. 2024-05-27T06:58:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 Processor -- start 2024-05-27T06:58:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 -- start start 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc072520 0x7f0dfc0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0ca7e0 con 0x7f0dfc0c9630 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0cd280 con 0x7f0dfc15dac0 2024-05-27T06:58:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e02600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc0ca960 con 0x7f0dfc072520 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45858/0 (socket says 172.21.15.81:45858) 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 -- 172.21.15.81:0/2033264434 learned_addr learned my addr 172.21.15.81:0/2033264434 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e01200640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc072520 0x7f0dfc0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0e01c00640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 -- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc072520 msgr2=0x7f0dfc0c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc072520 0x7f0dfc0c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 -- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dfc15dac0 msgr2=0x7f0dfc14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.558+0000 7f0df3e00640 1 -- 172.21.15.81:0/2033264434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc1f7370 con 0x7f0dfc0c9630 2024-05-27T06:58:10.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df3e00640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0de401ab40 tx=0x7f0de4019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40bb070 con 0x7f0dfc0c9630 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de40addf0 con 0x7f0dfc0c9630 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40b2040 con 0x7f0dfc0c9630 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc0c4c10 con 0x7f0dfc0c9630 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc0c6380 con 0x7f0dfc0c9630 2024-05-27T06:58:10.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc050870 con 0x7f0dfc0c9630 2024-05-27T06:58:10.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0de40be070 con 0x7f0dfc0c9630 2024-05-27T06:58:10.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 0x7f0dd8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:10.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f0de41285f0 con 0x7f0dfc0c9630 2024-05-27T06:58:10.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.562+0000 7f0e01200640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 0x7f0dd8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:10.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.566+0000 7f0e01200640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 0x7f0dd8071910 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0c6cb0 tx=0x7f0dec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:10.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.570+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0de4016110 con 0x7f0dfc0c9630 2024-05-27T06:58:10.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.762+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dfc15a130 con 0x7f0dfc0c9630 2024-05-27T06:58:10.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.762+0000 7f0df9600640 1 -- 172.21.15.81:0/2033264434 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f0de40f5400 con 0x7f0dfc0c9630 2024-05-27T06:58:10.754 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:10.754 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:10.755 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.762+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 msgr2=0x7f0dd8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.762+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 0x7f0dd8071910 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0c6cb0 tx=0x7f0dec008040 comp rx=0 tx=0).stop 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 msgr2=0x7f0dfc0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0de401ab40 tx=0x7f0de4019230 comp rx=0 tx=0).stop 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 shutdown_connections 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0dd806f400 0x7f0dd8071910 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0dfc15dac0 0x7f0dfc14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dfc0c9630 0x7f0dfc0ce990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 --2- 172.21.15.81:0/2033264434 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0dfc072520 0x7f0dfc0c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:10.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 >> 172.21.15.81:0/2033264434 conn(0x7f0dfc068880 msgr2=0x7f0dfc152b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:10.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 shutdown_connections 2024-05-27T06:58:10.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:10.766+0000 7f0e02600640 1 -- 172.21.15.81:0/2033264434 wait complete. 2024-05-27T06:58:10.775 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:11.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~128s 2024-05-27T06:58:12.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 Processor -- start 2024-05-27T06:58:12.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 -- start start 2024-05-27T06:58:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2ce00d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2ce0154200 0x7f2ce0154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ce00c78b0 con 0x7f2ce00ca770 2024-05-27T06:58:12.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f2ce0154b50 con 0x7f2ce0154200 2024-05-27T06:58:12.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2ce6400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce00c6ee0 con 0x7f2ce00cfdb0 2024-05-27T06:58:12.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2ce00ca770 0x7f2ce00cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38662/0 (socket says 172.21.15.81:38662) 2024-05-27T06:58:12.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 -- 172.21.15.81:0/2763590100 learned_addr learned my addr 172.21.15.81:0/2763590100 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdea00640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2ce00d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/2763590100 >> v2:172.21.15.106:3300/0 conn(0x7f2ce0154200 0x7f2ce0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415223847 0 0) 0x7f2ce00c78b0 con 0x7f2ce00ca770 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cc4003530 con 0x7f2ce00ca770 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 375554601 0 0) 0x7f2cc4003530 con 0x7f2ce00ca770 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 msgr2=0x7f2ce00d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2ce00d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 >> v2:172.21.15.106:3300/0 conn(0x7f2ce0154200 msgr2=0x7f2ce0154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 --2- 172.21.15.81:0/2763590100 >> v2:172.21.15.106:3300/0 conn(0x7f2ce0154200 0x7f2ce0154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce00c71c0 con 0x7f2ce00ca770 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f2cd0001700 con 0x7f2ce00ca770 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 >> v1:172.21.15.81:6789/0 conn(0x7f2ce00ca770 legacy=0x7f2ce00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cc4003a10 0x7f2cc4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2cc4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4004d50 0x7f2cc4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce00c6ee0 con 0x7f2cc4003a10 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc400ba80 con 0x7f2cc4004d50 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc400bc00 con 0x7f2ce00cfdb0 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cc4003a10 0x7f2cc4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdea00640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2cc4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4004d50 0x7f2cc4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 -- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 msgr2=0x7f2cc4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:12.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2cc4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 -- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4004d50 msgr2=0x7f2cc4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4004d50 0x7f2cc4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.946+0000 7f2cdf400640 1 -- 172.21.15.81:0/2763590100 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc4003530 con 0x7f2cc4003a10 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cdea00640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce00cfdb0 0x7f2cc4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4004d50 0x7f2cc4005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cdf400640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cc4003a10 0x7f2cc4003e80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c210 tx=0x7f2cd001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa4fce6131d7d6e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:12.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0007d40 con 0x7f2cc4003a10 2024-05-27T06:58:12.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd00a9020 con 0x7f2cc4003a10 2024-05-27T06:58:12.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cde000640 1 -- 172.21.15.81:0/2763590100 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0005b00 con 0x7f2cc4003a10 2024-05-27T06:58:12.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 -- 172.21.15.81:0/2763590100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cc4003a10 msgr2=0x7f2cc4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:12.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 --2- 172.21.15.81:0/2763590100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cc4003a10 0x7f2cc4003e80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c210 tx=0x7f2cd001cc00 comp rx=0 tx=0).stop 2024-05-27T06:58:12.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2cdfe00640 1 -- 172.21.15.81:0/2763590100 reap_dead start 2024-05-27T06:58:12.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 -- 172.21.15.81:0/2763590100 shutdown_connections 2024-05-27T06:58:12.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 -- 172.21.15.81:0/2763590100 >> 172.21.15.81:0/2763590100 conn(0x7f2ce0068ae0 msgr2=0x7f2ce0068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:12.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 -- 172.21.15.81:0/2763590100 shutdown_connections 2024-05-27T06:58:12.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.950+0000 7f2ce6400640 1 -- 172.21.15.81:0/2763590100 wait complete. 2024-05-27T06:58:12.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 Processor -- start 2024-05-27T06:58:12.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 -- start start 2024-05-27T06:58:12.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4003a10 0x7f2ce015ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce006d8a0 0x7f2ce01f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce00c7fd0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce00c7e50 con 0x7f2cc4003a10 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2ce6400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2ce00cbed0 con 0x7f2ce006d8a0 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2cdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2cdfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45888/0 (socket says 172.21.15.81:45888) 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.954+0000 7f2cdfe00640 1 -- 172.21.15.81:0/3465706935 learned_addr learned my addr 172.21.15.81:0/3465706935 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdf400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4003a10 0x7f2ce015ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdea00640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce006d8a0 0x7f2ce01f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 -- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce006d8a0 msgr2=0x7f2ce01f6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:12.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce006d8a0 0x7f2ce01f6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 -- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4003a10 msgr2=0x7f2ce015ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:12.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4003a10 0x7f2ce015ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:12.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 -- 172.21.15.81:0/3465706935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce015f5f0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cdfe00640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00c93d0 tx=0x7f2cd8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8010d80 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd80039a0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8003dd0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce015f8d0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce01f6770 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce006a560 con 0x7f2ce014ffe0 2024-05-27T06:58:12.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2cd801e020 con 0x7f2ce014ffe0 2024-05-27T06:58:12.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.958+0000 7f2cd7400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 0x7f2cc0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:12.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.962+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f2cd808fbc0 con 0x7f2ce014ffe0 2024-05-27T06:58:12.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.962+0000 7f2cdf400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 0x7f2cc0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:12.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.962+0000 7f2cdf400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 0x7f2cc0071940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c210 tx=0x7f2ce00ca680 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:12.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:12.966+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2cd805c820 con 0x7f2ce014ffe0 2024-05-27T06:58:13.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.154+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ce00769b0 con 0x7f2ce014ffe0 2024-05-27T06:58:13.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.154+0000 7f2cd7400640 1 -- 172.21.15.81:0/3465706935 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f2cd805c120 con 0x7f2ce014ffe0 2024-05-27T06:58:13.147 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:13.147 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:13.147 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 msgr2=0x7f2cc0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 0x7f2cc0071940 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c210 tx=0x7f2ce00ca680 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 msgr2=0x7f2ce0159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00c93d0 tx=0x7f2cd8010590 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 shutdown_connections 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ce006d8a0 0x7f2ce01f6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2cc006f430 0x7f2cc0071940 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce014ffe0 0x7f2ce0159a60 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 --2- 172.21.15.81:0/3465706935 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cc4003a10 0x7f2ce015ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 >> 172.21.15.81:0/3465706935 conn(0x7f2ce0068ae0 msgr2=0x7f2ce01532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:13.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 shutdown_connections 2024-05-27T06:58:13.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:13.158+0000 7f2ce6400640 1 -- 172.21.15.81:0/3465706935 wait complete. 2024-05-27T06:58:13.169 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:15.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:15.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.337+0000 7f71d6600640 1 Processor -- start 2024-05-27T06:58:15.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.337+0000 7f71d6600640 1 -- start start 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71d6600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f71d015ac20 0x7f71d00c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71d6600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d015db00 con 0x7f71d00c6fd0 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71d6600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d0073860 con 0x7f71d01546f0 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71d6600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f71d015e6f0 con 0x7f71d015ac20 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71c6a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f71d00c6fd0 0x7f71d015a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38664/0 (socket says 172.21.15.81:38664) 2024-05-27T06:58:15.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71c6a00640 1 -- 172.21.15.81:0/3444782316 learned_addr learned my addr 172.21.15.81:0/3444782316 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cfe00640 1 --2- 172.21.15.81:0/3444782316 >> v2:172.21.15.81:3300/0 conn(0x7f71d015ac20 0x7f71d00c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cea00640 1 -- 172.21.15.81:0/3444782316 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801984735 0 0) 0x7f71d015db00 con 0x7f71d00c6fd0 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cea00640 1 -- 172.21.15.81:0/3444782316 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b4003530 con 0x7f71d00c6fd0 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cfe00640 1 -- 172.21.15.81:0/3444782316 >> v1:172.21.15.106:6789/0 conn(0x7f71d01546f0 legacy=0x7f71d0072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cfe00640 1 -- 172.21.15.81:0/3444782316 >> v1:172.21.15.81:6789/0 conn(0x7f71d00c6fd0 legacy=0x7f71d015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.341+0000 7f71cfe00640 1 -- 172.21.15.81:0/3444782316 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d015ddc0 con 0x7f71d015ac20 2024-05-27T06:58:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71cfe00640 1 --2- 172.21.15.81:0/3444782316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d015ac20 0x7f71d00c73e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f71c80097c0 tx=0x7f71c801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfbfd581d7cec16 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71cea00640 1 -- 172.21.15.81:0/3444782316 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c80b5070 con 0x7f71d015ac20 2024-05-27T06:58:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71cea00640 1 -- 172.21.15.81:0/3444782316 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c80a8990 con 0x7f71d015ac20 2024-05-27T06:58:15.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71cea00640 1 -- 172.21.15.81:0/3444782316 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c80a8d40 con 0x7f71d015ac20 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- 172.21.15.81:0/3444782316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d015ac20 msgr2=0x7f71d00c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 --2- 172.21.15.81:0/3444782316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d015ac20 0x7f71d00c73e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f71c80097c0 tx=0x7f71c801ec90 comp rx=0 tx=0).stop 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- 172.21.15.81:0/3444782316 shutdown_connections 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 --2- 172.21.15.81:0/3444782316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d015ac20 0x7f71d00c73e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- 172.21.15.81:0/3444782316 >> 172.21.15.81:0/3444782316 conn(0x7f71d0070700 msgr2=0x7f71d00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:15.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- 172.21.15.81:0/3444782316 shutdown_connections 2024-05-27T06:58:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- 172.21.15.81:0/3444782316 wait complete. 2024-05-27T06:58:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 Processor -- start 2024-05-27T06:58:15.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.345+0000 7f71d6600640 1 -- start start 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d00c6fd0 0x7f71d01f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 0x7f71d014b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71d015e380 con 0x7f71d01546f0 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f71d00755b0 con 0x7f71d015ac20 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f71d0154b00 con 0x7f71d00c6fd0 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45902/0 (socket says 172.21.15.81:45902) 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 -- 172.21.15.81:0/1086877560 learned_addr learned my addr 172.21.15.81:0/1086877560 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71cfe00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 0x7f71d014b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:15.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71cf400640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d00c6fd0 0x7f71d01f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 -- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d00c6fd0 msgr2=0x7f71d01f1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d00c6fd0 0x7f71d01f1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 -- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 msgr2=0x7f71d014b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 0x7f71d014b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 -- 172.21.15.81:0/1086877560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d00cd180 con 0x7f71d01546f0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71cfe00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 0x7f71d014b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c6a00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c390 tx=0x7f71b800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8017070 con 0x7f71d01546f0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b8005cf0 con 0x7f71d01546f0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b8007310 con 0x7f71d01546f0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d01f1560 con 0x7f71d01546f0 2024-05-27T06:58:15.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d0155110 con 0x7f71d01546f0 2024-05-27T06:58:15.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.349+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d006a560 con 0x7f71d01546f0 2024-05-27T06:58:15.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.353+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f71b801a070 con 0x7f71d01546f0 2024-05-27T06:58:15.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.353+0000 7f71c7e00640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 0x7f71ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:15.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.353+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f71b808e8c0 con 0x7f71d01546f0 2024-05-27T06:58:15.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.353+0000 7f71cf400640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 0x7f71ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:15.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.353+0000 7f71cf400640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 0x7f71ac071940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f71c0000f90 tx=0x7f71c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:15.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.357+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f71b805b520 con 0x7f71d01546f0 2024-05-27T06:58:15.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.549+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71d0154ea0 con 0x7f71d01546f0 2024-05-27T06:58:15.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.549+0000 7f71c7e00640 1 -- 172.21.15.81:0/1086877560 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f71b805ae20 con 0x7f71d01546f0 2024-05-27T06:58:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:15.543 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:15.543 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 msgr2=0x7f71ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 0x7f71ac071940 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f71c0000f90 tx=0x7f71c0009040 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 msgr2=0x7f71d00ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f71b800c390 tx=0x7f71b800c850 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 shutdown_connections 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f71ac06f430 0x7f71ac071940 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f71d015ac20 0x7f71d014b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71d01546f0 0x7f71d00ca590 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 --2- 172.21.15.81:0/1086877560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f71d00c6fd0 0x7f71d01f1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 >> 172.21.15.81:0/1086877560 conn(0x7f71d0070700 msgr2=0x7f71d00c9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 shutdown_connections 2024-05-27T06:58:15.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:15.553+0000 7f71d6600640 1 -- 172.21.15.81:0/1086877560 wait complete. 2024-05-27T06:58:15.565 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:17.566 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:17.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.733+0000 7ff6b7200640 1 Processor -- start 2024-05-27T06:58:17.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.733+0000 7ff6b7200640 1 -- start start 2024-05-27T06:58:17.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff6b0072430 0x7ff6b0159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 0x7ff6b00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 0x7ff6b015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff6b00c5040 con 0x7ff6b0072430 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff6b00c75c0 con 0x7ff6b015a040 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b7200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b00c7dd0 con 0x7ff6b00c60a0 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff6b0072430 0x7ff6b0159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff6b0072430 0x7ff6b0159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53948/0 (socket says 172.21.15.81:53948) 2024-05-27T06:58:17.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 -- 172.21.15.81:0/3986771276 learned_addr learned my addr 172.21.15.81:0/3986771276 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6a7400640 1 --2- 172.21.15.81:0/3986771276 >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 0x7ff6b00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b5200640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 0x7ff6b015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 -- 172.21.15.81:0/3986771276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 msgr2=0x7ff6b015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 0x7ff6b015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 -- 172.21.15.81:0/3986771276 >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 msgr2=0x7ff6b00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 --2- 172.21.15.81:0/3986771276 >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 0x7ff6b00c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6afe00640 1 -- 172.21.15.81:0/3986771276 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b0072840 con 0x7ff6b0072430 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6a7400640 1 --2- 172.21.15.81:0/3986771276 >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 0x7ff6b00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:17.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.737+0000 7ff6b5200640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 0x7ff6b015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:17.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6afe00640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b0072430 0x7ff6b0159b00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff6b006f550 tx=0x7ff6a001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812d49fc83f5e889 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:17.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6af400640 1 -- 172.21.15.81:0/3986771276 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a00b0070 con 0x7ff6b0072430 2024-05-27T06:58:17.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6af400640 1 -- 172.21.15.81:0/3986771276 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a00ab600 con 0x7ff6b0072430 2024-05-27T06:58:17.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6af400640 1 -- 172.21.15.81:0/3986771276 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a00ab9d0 con 0x7ff6b0072430 2024-05-27T06:58:17.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 -- 172.21.15.81:0/3986771276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b0072430 msgr2=0x7ff6b0159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:17.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b0072430 0x7ff6b0159b00 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff6b006f550 tx=0x7ff6a001c260 comp rx=0 tx=0).stop 2024-05-27T06:58:17.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 -- 172.21.15.81:0/3986771276 shutdown_connections 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b00c60a0 0x7ff6b015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 --2- 172.21.15.81:0/3986771276 >> v2:172.21.15.106:3300/0 conn(0x7ff6b015a040 0x7ff6b00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 --2- 172.21.15.81:0/3986771276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b0072430 0x7ff6b0159b00 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 -- 172.21.15.81:0/3986771276 >> 172.21.15.81:0/3986771276 conn(0x7ff6b0070700 msgr2=0x7ff6b00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 -- 172.21.15.81:0/3986771276 shutdown_connections 2024-05-27T06:58:17.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.741+0000 7ff6b7200640 1 -- 172.21.15.81:0/3986771276 wait complete. 2024-05-27T06:58:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 Processor -- start 2024-05-27T06:58:17.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- start start 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b0072430 0x7ff6b01f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 0x7ff6b014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b00ce6f0 con 0x7ff6b015a040 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01595e0 con 0x7ff6b00c60a0 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b00c5610 con 0x7ff6b0072430 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53954/0 (socket says 172.21.15.81:53954) 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 -- 172.21.15.81:0/1020143130 learned_addr learned my addr 172.21.15.81:0/1020143130 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6a7400640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 0x7ff6b014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 -- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b0072430 msgr2=0x7ff6b01f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b0072430 0x7ff6b01f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 -- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 msgr2=0x7ff6b014df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 0x7ff6b014df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 -- 172.21.15.81:0/1020143130 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b01f6410 con 0x7ff6b015a040 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6a7400640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 0x7ff6b014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:17.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b5200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800cca0 tx=0x7ff6a800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800f680 con 0x7ff6b015a040 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a800f840 con 0x7ff6b015a040 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b0158a10 con 0x7ff6b015a040 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800fc30 con 0x7ff6b015a040 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b0154dc0 con 0x7ff6b015a040 2024-05-27T06:58:17.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.745+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b01f65e0 con 0x7ff6b015a040 2024-05-27T06:58:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.749+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff6a80213f0 con 0x7ff6b015a040 2024-05-27T06:58:17.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.749+0000 7ff6acc00640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 0x7ff68c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.749+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff6a8090260 con 0x7ff6b015a040 2024-05-27T06:58:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.749+0000 7ff6afe00640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 0x7ff68c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:17.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.749+0000 7ff6afe00640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 0x7ff68c071910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00ad000 tx=0x7ff6a0004740 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:17.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.753+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff6a8014030 con 0x7ff6b015a040 2024-05-27T06:58:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6b006a560 con 0x7ff6b015a040 2024-05-27T06:58:17.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6acc00640 1 -- 172.21.15.81:0/1020143130 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff6a805cec0 con 0x7ff6b015a040 2024-05-27T06:58:17.937 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:17.938 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:17.938 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:17.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 msgr2=0x7ff68c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:17.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 0x7ff68c071910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00ad000 tx=0x7ff6a0004740 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 msgr2=0x7ff6b014b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800cca0 tx=0x7ff6a800a650 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 shutdown_connections 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff68c06f400 0x7ff68c071910 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6b015a040 0x7ff6b014b1d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6b00c60a0 0x7ff6b014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 --2- 172.21.15.81:0/1020143130 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff6b0072430 0x7ff6b01f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 >> 172.21.15.81:0/1020143130 conn(0x7ff6b0070700 msgr2=0x7ff6b0074900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.945+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 shutdown_connections 2024-05-27T06:58:17.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:17.949+0000 7ff6b7200640 1 -- 172.21.15.81:0/1020143130 wait complete. 2024-05-27T06:58:17.962 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:58:18.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~134s 2024-05-27T06:58:19.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:20.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 Processor -- start 2024-05-27T06:58:20.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 -- start start 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 0x7fc7141640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc714072840 con 0x7fc714072430 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc7140c5040 con 0x7fc714161cd0 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc71aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc7140c75c0 con 0x7fc7140c5b60 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc718a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 0x7fc7141640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:52016/0 (socket says 172.21.15.81:52016) 2024-05-27T06:58:20.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 -- 172.21.15.81:0/2120496238 learned_addr learned my addr 172.21.15.81:0/2120496238 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:20.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 -- 172.21.15.81:0/2120496238 >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 msgr2=0x7fc7141640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:20.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 --2- 172.21.15.81:0/2120496238 >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 0x7fc7141640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 -- 172.21.15.81:0/2120496238 >> v1:172.21.15.106:6789/0 conn(0x7fc714072430 legacy=0x7fc714159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc70b400640 1 -- 172.21.15.81:0/2120496238 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7140c5270 con 0x7fc7140c5b60 2024-05-27T06:58:20.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.129+0000 7fc718a00640 1 --2- 172.21.15.81:0/2120496238 >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 0x7fc7141640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:58:20.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc70b400640 1 --2- 172.21.15.81:0/2120496238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009870 tx=0x7fc6fc01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca498ee11d63d006 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:20.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc713400640 1 -- 172.21.15.81:0/2120496238 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0b2070 con 0x7fc7140c5b60 2024-05-27T06:58:20.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc713400640 1 -- 172.21.15.81:0/2120496238 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0a9070 con 0x7fc7140c5b60 2024-05-27T06:58:20.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc713400640 1 -- 172.21.15.81:0/2120496238 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0ad880 con 0x7fc7140c5b60 2024-05-27T06:58:20.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 -- 172.21.15.81:0/2120496238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 msgr2=0x7fc7140c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 --2- 172.21.15.81:0/2120496238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009870 tx=0x7fc6fc01c560 comp rx=0 tx=0).stop 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 -- 172.21.15.81:0/2120496238 shutdown_connections 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 --2- 172.21.15.81:0/2120496238 >> v2:172.21.15.106:3300/0 conn(0x7fc714161cd0 0x7fc7141640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 --2- 172.21.15.81:0/2120496238 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc7140c5f70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 -- 172.21.15.81:0/2120496238 >> 172.21.15.81:0/2120496238 conn(0x7fc714070700 msgr2=0x7fc7140cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 -- 172.21.15.81:0/2120496238 shutdown_connections 2024-05-27T06:58:20.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.133+0000 7fc71aa00640 1 -- 172.21.15.81:0/2120496238 wait complete. 2024-05-27T06:58:20.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 Processor -- start 2024-05-27T06:58:20.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 -- start start 2024-05-27T06:58:20.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc714072430 0x7fc7141f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc714157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7140ce6f0 con 0x7fc714161cd0 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc7140c7dd0 con 0x7fc714072430 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc71aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc7140c5610 con 0x7fc7140c5b60 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc718a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc718a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53956/0 (socket says 172.21.15.81:53956) 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc714157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc718a00640 1 -- 172.21.15.81:0/1323140315 learned_addr learned my addr 172.21.15.81:0/1323140315 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:20.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc713e00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc714072430 0x7fc7141f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 -- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc714072430 msgr2=0x7fc7141f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc714072430 0x7fc7141f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 -- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 msgr2=0x7fc71414dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc70b400640 1 -- 172.21.15.81:0/1323140315 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7141f6ea0 con 0x7fc7140c5b60 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.137+0000 7fc718a00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc70b400640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc714157d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc7140c7740 tx=0x7fc6fc019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0bb070 con 0x7fc7140c5b60 2024-05-27T06:58:20.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0add80 con 0x7fc7140c5b60 2024-05-27T06:58:20.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7141582b0 con 0x7fc7140c5b60 2024-05-27T06:58:20.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0b2040 con 0x7fc7140c5b60 2024-05-27T06:58:20.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc714166fd0 con 0x7fc7140c5b60 2024-05-27T06:58:20.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71406a560 con 0x7fc7140c5b60 2024-05-27T06:58:20.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.141+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc6fc0be070 con 0x7fc7140c5b60 2024-05-27T06:58:20.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.145+0000 7fc710c00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 0x7fc6f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:20.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.145+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fc6fc128250 con 0x7fc7140c5b60 2024-05-27T06:58:20.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.145+0000 7fc713e00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 0x7fc6f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:20.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.145+0000 7fc713e00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 0x7fc6f0071910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc704007fd0 tx=0x7fc704009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:20.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.145+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc6fc016150 con 0x7fc7140c5b60 2024-05-27T06:58:20.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7140c5940 con 0x7fc7140c5b60 2024-05-27T06:58:20.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc710c00640 1 -- 172.21.15.81:0/1323140315 <== mon.2 v2:172.21.15.106:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fc6fc0f49f0 con 0x7fc7140c5b60 2024-05-27T06:58:20.329 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:20.329 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:20.329 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 msgr2=0x7fc6f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 0x7fc6f0071910 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc704007fd0 tx=0x7fc704009040 comp rx=0 tx=0).stop 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 msgr2=0x7fc714157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc714157d70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc7140c7740 tx=0x7fc6fc019300 comp rx=0 tx=0).stop 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 shutdown_connections 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc7140c5b60 0x7fc714157d70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc6f006f400 0x7fc6f0071910 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc714161cd0 0x7fc71414dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 --2- 172.21.15.81:0/1323140315 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc714072430 0x7fc7141f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:20.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 >> 172.21.15.81:0/1323140315 conn(0x7fc714070700 msgr2=0x7fc714076080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:20.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 shutdown_connections 2024-05-27T06:58:20.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:20.337+0000 7fc71aa00640 1 -- 172.21.15.81:0/1323140315 wait complete. 2024-05-27T06:58:20.351 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:22.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:22.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 Processor -- start 2024-05-27T06:58:22.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 -- start start 2024-05-27T06:58:22.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2b4c0cfdb0 0x7f2b4c0d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:22.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4c0c78b0 con 0x7f2b4c0ca770 2024-05-27T06:58:22.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4c0c6ee0 con 0x7f2b4c154200 2024-05-27T06:58:22.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b51800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4c0c71c0 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b4aa00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2b4c0ca770 0x7f2b4c0cb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:54484/0 (socket says 172.21.15.81:54484) 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b4aa00640 1 -- 172.21.15.81:0/595841157 learned_addr learned my addr 172.21.15.81:0/595841157 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.521+0000 7f2b42000640 1 --2- 172.21.15.81:0/595841157 >> v2:172.21.15.106:3300/0 conn(0x7f2b4c0cfdb0 0x7f2b4c0d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b4a000640 1 -- 172.21.15.81:0/595841157 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2266324251 0 0) 0x7f2b4c0c78b0 con 0x7f2b4c0ca770 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b4a000640 1 -- 172.21.15.81:0/595841157 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b30003530 con 0x7f2b4c0ca770 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b42000640 1 -- 172.21.15.81:0/595841157 >> v1:172.21.15.106:6789/0 conn(0x7f2b4c154200 legacy=0x7f2b4c164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b42000640 1 -- 172.21.15.81:0/595841157 >> v1:172.21.15.81:6789/0 conn(0x7f2b4c0ca770 legacy=0x7f2b4c0cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b42000640 1 -- 172.21.15.81:0/595841157 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c0c7b70 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b42000640 1 --2- 172.21.15.81:0/595841157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0cfdb0 0x7f2b4c0d01c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b3401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d23daba654c619b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:22.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b4a000640 1 -- 172.21.15.81:0/595841157 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b340b1070 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b4a000640 1 -- 172.21.15.81:0/595841157 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b340ac410 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.525+0000 7f2b4a000640 1 -- 172.21.15.81:0/595841157 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b340ac7c0 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- 172.21.15.81:0/595841157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0cfdb0 msgr2=0x7f2b4c0d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 --2- 172.21.15.81:0/595841157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0cfdb0 0x7f2b4c0d01c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097c0 tx=0x7f2b3401c4a0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- 172.21.15.81:0/595841157 shutdown_connections 2024-05-27T06:58:22.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 --2- 172.21.15.81:0/595841157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0cfdb0 0x7f2b4c0d01c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- 172.21.15.81:0/595841157 >> 172.21.15.81:0/595841157 conn(0x7f2b4c068ae0 msgr2=0x7f2b4c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:22.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- 172.21.15.81:0/595841157 shutdown_connections 2024-05-27T06:58:22.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- 172.21.15.81:0/595841157 wait complete. 2024-05-27T06:58:22.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 Processor -- start 2024-05-27T06:58:22.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.529+0000 7f2b51800640 1 -- start start 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0ca770 0x7f2b4c152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b4c0cfdb0 0x7f2b4c1f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0cc050 con 0x7f2b4c154200 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0cbed0 con 0x7f2b4c0ca770 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c168cb0 con 0x7f2b4c0cfdb0 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4aa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0ca770 0x7f2b4c152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53962/0 (socket says 172.21.15.81:53962) 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 -- 172.21.15.81:0/215581240 learned_addr learned my addr 172.21.15.81:0/215581240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:22.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b42000640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b4c0cfdb0 0x7f2b4c1f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:22.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 -- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b4c0cfdb0 msgr2=0x7f2b4c1f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b4c0cfdb0 0x7f2b4c1f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 -- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0ca770 msgr2=0x7f2b4c152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0ca770 0x7f2b4c152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 -- 172.21.15.81:0/215581240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c1f67d0 con 0x7f2b4c154200 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b4b400640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400c740 tx=0x7f2b4400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44003b80 con 0x7f2b4c154200 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b4400ce50 con 0x7f2b4c154200 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c1517d0 con 0x7f2b4c154200 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4400a850 con 0x7f2b4c154200 2024-05-27T06:58:22.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c151d50 con 0x7f2b4c154200 2024-05-27T06:58:22.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.533+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c06a560 con 0x7f2b4c154200 2024-05-27T06:58:22.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.537+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2b4400a9f0 con 0x7f2b4c154200 2024-05-27T06:58:22.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.537+0000 7f2b43400640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 0x7f2b28071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:22.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.537+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f2b4408f330 con 0x7f2b4c154200 2024-05-27T06:58:22.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.537+0000 7f2b4aa00640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 0x7f2b28071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:22.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.537+0000 7f2b4aa00640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 0x7f2b28071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c005890 tx=0x7f2b3c007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:22.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.541+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2b4405b8c0 con 0x7f2b4c154200 2024-05-27T06:58:22.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b4c1538f0 con 0x7f2b4c154200 2024-05-27T06:58:22.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b43400640 1 -- 172.21.15.81:0/215581240 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f2b4405fbc0 con 0x7f2b4c154200 2024-05-27T06:58:22.729 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:22.729 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:22.729 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 msgr2=0x7f2b28071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 0x7f2b28071910 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c005890 tx=0x7f2b3c007ce0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 msgr2=0x7f2b4c14d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400c740 tx=0x7f2b4400cc00 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 shutdown_connections 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2b4c0cfdb0 0x7f2b4c1f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2b2806f400 0x7f2b28071910 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4c154200 0x7f2b4c14d550 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 --2- 172.21.15.81:0/215581240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b4c0ca770 0x7f2b4c152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 >> 172.21.15.81:0/215581240 conn(0x7f2b4c068ae0 msgr2=0x7f2b4c15e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:22.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 shutdown_connections 2024-05-27T06:58:22.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:22.737+0000 7f2b51800640 1 -- 172.21.15.81:0/215581240 wait complete. 2024-05-27T06:58:22.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:24.752 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:24.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.917+0000 7f27a0000640 1 Processor -- start 2024-05-27T06:58:24.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.917+0000 7f27a0000640 1 -- start start 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f27a0000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f279815ac20 0x7f27980c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f27a0000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279815db00 con 0x7f27980c6fd0 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f27a0000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2798073860 con 0x7f27981546f0 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f27a0000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f279815e6f0 con 0x7f279815ac20 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279cc00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f27980c6fd0 0x7f279815a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:54492/0 (socket says 172.21.15.81:54492) 2024-05-27T06:58:24.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279cc00640 1 -- 172.21.15.81:0/3350836309 learned_addr learned my addr 172.21.15.81:0/3350836309 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:24.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279e000640 1 --2- 172.21.15.81:0/3350836309 >> v2:172.21.15.106:3300/0 conn(0x7f279815ac20 0x7f27980c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317185569 0 0) 0x7f279815db00 con 0x7f27980c6fd0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2780003530 con 0x7f27980c6fd0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1589174721 0 0) 0x7f2780003530 con 0x7f27980c6fd0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 >> v2:172.21.15.106:3300/0 conn(0x7f279815ac20 msgr2=0x7f27980c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 --2- 172.21.15.81:0/3350836309 >> v2:172.21.15.106:3300/0 conn(0x7f279815ac20 0x7f27980c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 >> v1:172.21.15.106:6789/0 conn(0x7f27981546f0 legacy=0x7f2798072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279815ddc0 con 0x7f27980c6fd0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f2784001700 con 0x7f27980c6fd0 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 >> v1:172.21.15.81:6789/0 conn(0x7f27980c6fd0 legacy=0x7f279815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f2780003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780004810 0x7f2780004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2780005260 0x7f278000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f279815e6f0 con 0x7f2780003a60 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f278000c1e0 con 0x7f2780004810 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f278000c3e0 con 0x7f2780005260 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f2780003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279e000640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2780005260 0x7f278000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279cc00640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780004810 0x7f2780004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 -- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2780005260 msgr2=0x7f278000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2780005260 0x7f278000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 -- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780004810 msgr2=0x7f2780004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:24.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780004810 0x7f2780004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279d600640 1 -- 172.21.15.81:0/3350836309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780003530 con 0x7f2780003a60 2024-05-27T06:58:24.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279cc00640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2780004810 0x7f2780004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:24.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.921+0000 7f279e000640 1 -- 172.21.15.81:0/3350836309 reap_dead start 2024-05-27T06:58:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f279d600640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f2780003ed0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f278c001e90 tx=0x7f278c01c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a41f70858a6338 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c001ae0 con 0x7f2780003a60 2024-05-27T06:58:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278c003b80 con 0x7f2780003a60 2024-05-27T06:58:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f2797e00640 1 -- 172.21.15.81:0/3350836309 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c005080 con 0x7f2780003a60 2024-05-27T06:58:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 -- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 msgr2=0x7f2780003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f2780003ed0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f278c001e90 tx=0x7f278c01c710 comp rx=0 tx=0).stop 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 -- 172.21.15.81:0/3350836309 shutdown_connections 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 --2- 172.21.15.81:0/3350836309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f2780003ed0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 -- 172.21.15.81:0/3350836309 >> 172.21.15.81:0/3350836309 conn(0x7f2798070700 msgr2=0x7f27980cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 -- 172.21.15.81:0/3350836309 shutdown_connections 2024-05-27T06:58:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.925+0000 7f27a0000640 1 -- 172.21.15.81:0/3350836309 wait complete. 2024-05-27T06:58:24.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 Processor -- start 2024-05-27T06:58:24.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 -- start start 2024-05-27T06:58:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f279814e6b0 0x7f279814dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27981f3410 0x7f27981f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f279815e380 con 0x7f2780003a60 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27980755b0 con 0x7f27981f3410 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f27a0000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2798154b00 con 0x7f279814e6b0 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f279d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f279d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53986/0 (socket says 172.21.15.81:53986) 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f279d600640 1 -- 172.21.15.81:0/802193112 learned_addr learned my addr 172.21.15.81:0/802193112 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f279cc00640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f279814e6b0 0x7f279814dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.929+0000 7f279e000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27981f3410 0x7f27981f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 -- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f279814e6b0 msgr2=0x7f279814dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f279814e6b0 0x7f279814dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 -- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27981f3410 msgr2=0x7f27981f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27981f3410 0x7f27981f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 -- 172.21.15.81:0/802193112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279815d280 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f279d600640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f278c001e90 tx=0x7f2780005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c001ac0 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278c006b60 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279815bf80 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c0a6470 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279815c540 con 0x7f2780003a60 2024-05-27T06:58:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279806a560 con 0x7f2780003a60 2024-05-27T06:58:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.933+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f278c0a6610 con 0x7f2780003a60 2024-05-27T06:58:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.937+0000 7f2795600640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 0x7f2778071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.937+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f278c129880 con 0x7f2780003a60 2024-05-27T06:58:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.937+0000 7f279cc00640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 0x7f2778071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.937+0000 7f279cc00640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 0x7f2778071940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27981eb300 tx=0x7f2784004820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:24.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:24.941+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f278c0ac020 con 0x7f2780003a60 2024-05-27T06:58:25.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27980c8e20 con 0x7f2780003a60 2024-05-27T06:58:25.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f2795600640 1 -- 172.21.15.81:0/802193112 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f278c016470 con 0x7f2780003a60 2024-05-27T06:58:25.121 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:25.121 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:25.122 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:25.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 msgr2=0x7f2778071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 0x7f2778071940 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f27981eb300 tx=0x7f2784004820 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 msgr2=0x7f279815cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f278c001e90 tx=0x7f2780005840 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 shutdown_connections 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f279814e6b0 0x7f279814dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f277806f430 0x7f2778071940 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27981f3410 0x7f27981f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 --2- 172.21.15.81:0/802193112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2780003a60 0x7f279815cd40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.129+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 >> 172.21.15.81:0/802193112 conn(0x7f2798070700 msgr2=0x7f27980c9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.133+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 shutdown_connections 2024-05-27T06:58:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:25.133+0000 7f27a0000640 1 -- 172.21.15.81:0/802193112 wait complete. 2024-05-27T06:58:25.143 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:25.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~141s 2024-05-27T06:58:27.145 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:27.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 Processor -- start 2024-05-27T06:58:27.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 -- start start 2024-05-27T06:58:27.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 0x7f1ea015c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1ea00c96e0 0x7f1ea00c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea006c9a0 con 0x7f1ea0156b40 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea006db10 con 0x7f1ea00c96e0 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1ea5e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea006cc40 con 0x7f1ea00ca240 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e97e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1ea0156b40 0x7f1ea00cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34564/0 (socket says 172.21.15.81:34564) 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e97e00640 1 -- 172.21.15.81:0/3925192704 learned_addr learned my addr 172.21.15.81:0/3925192704 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3925192704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 0x7f1ea015c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9f400640 1 --2- 172.21.15.81:0/3925192704 >> v2:172.21.15.106:3300/0 conn(0x7f1ea00c96e0 0x7f1ea00c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9e000640 1 -- 172.21.15.81:0/3925192704 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 656079572 0 0) 0x7f1ea006c9a0 con 0x7f1ea0156b40 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9e000640 1 -- 172.21.15.81:0/3925192704 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e84003530 con 0x7f1ea0156b40 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9ea00640 1 -- 172.21.15.81:0/3925192704 >> v2:172.21.15.106:3300/0 conn(0x7f1ea00c96e0 msgr2=0x7f1ea00c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3925192704 >> v2:172.21.15.106:3300/0 conn(0x7f1ea00c96e0 0x7f1ea00c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9ea00640 1 -- 172.21.15.81:0/3925192704 >> v1:172.21.15.81:6789/0 conn(0x7f1ea0156b40 legacy=0x7f1ea00cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.313+0000 7f1e9ea00640 1 -- 172.21.15.81:0/3925192704 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea006cf00 con 0x7f1ea00ca240 2024-05-27T06:58:27.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3925192704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 0x7f1ea015c760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ea006cb70 tx=0x7f1e9001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35552ca000a1188 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:27.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1e9e000640 1 -- 172.21.15.81:0/3925192704 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e900b5070 con 0x7f1ea00ca240 2024-05-27T06:58:27.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1e9e000640 1 -- 172.21.15.81:0/3925192704 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e900a8990 con 0x7f1ea00ca240 2024-05-27T06:58:27.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1e9e000640 1 -- 172.21.15.81:0/3925192704 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e900a8d60 con 0x7f1ea00ca240 2024-05-27T06:58:27.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3925192704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 msgr2=0x7f1ea015c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3925192704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 0x7f1ea015c760 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1ea006cb70 tx=0x7f1e9001ec90 comp rx=0 tx=0).stop 2024-05-27T06:58:27.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3925192704 shutdown_connections 2024-05-27T06:58:27.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3925192704 >> v2:172.21.15.106:3300/0 conn(0x7f1ea00c96e0 0x7f1ea00c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3925192704 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00ca240 0x7f1ea015c760 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3925192704 >> 172.21.15.81:0/3925192704 conn(0x7f1ea00cb2c0 msgr2=0x7f1ea00cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:27.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3925192704 shutdown_connections 2024-05-27T06:58:27.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.317+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3925192704 wait complete. 2024-05-27T06:58:27.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 Processor -- start 2024-05-27T06:58:27.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 -- start start 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00c96e0 0x7f1ea0159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea00ca240 0x7f1ea014be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea006d7a0 con 0x7f1ea0156b40 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea006d4c0 con 0x7f1ea00ca240 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1ea5e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea006dfa0 con 0x7f1ea00c96e0 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.321+0000 7f1e9f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45670/0 (socket says 172.21.15.81:45670) 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 -- 172.21.15.81:0/3208934212 learned_addr learned my addr 172.21.15.81:0/3208934212 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00c96e0 0x7f1ea0159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e97e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea00ca240 0x7f1ea014be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 -- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00c96e0 msgr2=0x7f1ea0159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00c96e0 0x7f1ea0159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 -- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea00ca240 msgr2=0x7f1ea014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea00ca240 0x7f1ea014be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 -- 172.21.15.81:0/3208934212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea014c350 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e9f400640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800efc0 tx=0x7f1e9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98018020 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e98014480 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea00c7480 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e98009e60 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea00c7a40 con 0x7f1ea0156b40 2024-05-27T06:58:27.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea014c520 con 0x7f1ea0156b40 2024-05-27T06:58:27.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.325+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e9801b020 con 0x7f1ea0156b40 2024-05-27T06:58:27.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.329+0000 7f1e97400640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 0x7f1e7c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:27.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.329+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f1e98090170 con 0x7f1ea0156b40 2024-05-27T06:58:27.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.329+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 0x7f1e7c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:27.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.329+0000 7f1e9ea00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 0x7f1e7c071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1e900b2000 tx=0x7f1e90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.333+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1e98022020 con 0x7f1ea0156b40 2024-05-27T06:58:27.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea006a560 con 0x7f1ea0156b40 2024-05-27T06:58:27.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1e97400640 1 -- 172.21.15.81:0/3208934212 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f1e98010040 con 0x7f1ea0156b40 2024-05-27T06:58:27.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:27.514 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:27.514 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 msgr2=0x7f1e7c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 0x7f1e7c071910 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1e900b2000 tx=0x7f1e90008040 comp rx=0 tx=0).stop 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 msgr2=0x7f1ea00c69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.521+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800efc0 tx=0x7f1e9800c490 comp rx=0 tx=0).stop 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 shutdown_connections 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e7c06f400 0x7f1e7c071910 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea0156b40 0x7f1ea00c69c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ea00ca240 0x7f1ea014be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 --2- 172.21.15.81:0/3208934212 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1ea00c96e0 0x7f1ea0159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 >> 172.21.15.81:0/3208934212 conn(0x7f1ea00cb2c0 msgr2=0x7f1ea00742c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 shutdown_connections 2024-05-27T06:58:27.517 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:27.525+0000 7f1ea5e00640 1 -- 172.21.15.81:0/3208934212 wait complete. 2024-05-27T06:58:27.536 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:29.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:29.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 Processor -- start 2024-05-27T06:58:29.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 -- start start 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb6200c9850 0x7fb6200ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620071f20 0x7fb620072330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb62015c8d0 con 0x7fb620160040 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb62015d7a0 con 0x7fb6200c9850 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb626000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb62015cb90 con 0x7fb620071f20 2024-05-27T06:58:29.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb624c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb6200c9850 0x7fb6200ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb625600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb620160040 0x7fb620160450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34572/0 (socket says 172.21.15.81:34572) 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb625600640 1 -- 172.21.15.81:0/1214002983 learned_addr learned my addr 172.21.15.81:0/1214002983 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb61fe00640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620071f20 0x7fb620072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb61f400640 1 -- 172.21.15.81:0/1214002983 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585696755 0 0) 0x7fb62015c8d0 con 0x7fb620160040 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb61f400640 1 -- 172.21.15.81:0/1214002983 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb604003530 con 0x7fb620160040 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb624c00640 1 -- 172.21.15.81:0/1214002983 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620071f20 msgr2=0x7fb620072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb624c00640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620071f20 0x7fb620072330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb624c00640 1 -- 172.21.15.81:0/1214002983 >> v1:172.21.15.81:6789/0 conn(0x7fb620160040 legacy=0x7fb620160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.709+0000 7fb624c00640 1 -- 172.21.15.81:0/1214002983 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62015ce70 con 0x7fb6200c9850 2024-05-27T06:58:29.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb624c00640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6200c9850 0x7fb6200ca2e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8739d9de3137ea6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:29.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb61f400640 1 -- 172.21.15.81:0/1214002983 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100b2070 con 0x7fb6200c9850 2024-05-27T06:58:29.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb61f400640 1 -- 172.21.15.81:0/1214002983 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6100a5830 con 0x7fb6200c9850 2024-05-27T06:58:29.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb61f400640 1 -- 172.21.15.81:0/1214002983 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100a5c00 con 0x7fb6200c9850 2024-05-27T06:58:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 -- 172.21.15.81:0/1214002983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6200c9850 msgr2=0x7fb6200ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6200c9850 0x7fb6200ca2e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61001c970 comp rx=0 tx=0).stop 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 -- 172.21.15.81:0/1214002983 shutdown_connections 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620071f20 0x7fb620072330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 --2- 172.21.15.81:0/1214002983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6200c9850 0x7fb6200ca2e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.713+0000 7fb626000640 1 -- 172.21.15.81:0/1214002983 >> 172.21.15.81:0/1214002983 conn(0x7fb620068d90 msgr2=0x7fb6200cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- 172.21.15.81:0/1214002983 shutdown_connections 2024-05-27T06:58:29.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- 172.21.15.81:0/1214002983 wait complete. 2024-05-27T06:58:29.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 Processor -- start 2024-05-27T06:58:29.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- start start 2024-05-27T06:58:29.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6200c9850 0x7fb62015bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 0x7fb6200cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6200ca9a0 con 0x7fb620071f20 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb6200ca820 con 0x7fb6200c9850 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb626000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb6200c9c60 con 0x7fb620160040 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.717+0000 7fb624c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45694/0 (socket says 172.21.15.81:45694) 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 -- 172.21.15.81:0/3921754380 learned_addr learned my addr 172.21.15.81:0/3921754380 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:29.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb625600640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 0x7fb6200cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb61fe00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6200c9850 0x7fb62015bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 -- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 msgr2=0x7fb6200cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 0x7fb6200cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 -- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6200c9850 msgr2=0x7fb62015bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6200c9850 0x7fb62015bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 -- 172.21.15.81:0/3921754380 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62014d400 con 0x7fb620071f20 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb625600640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 0x7fb6200cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb624c00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb610019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100b2050 con 0x7fb620071f20 2024-05-27T06:58:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6100be070 con 0x7fb620071f20 2024-05-27T06:58:29.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100ad850 con 0x7fb620071f20 2024-05-27T06:58:29.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6201f6200 con 0x7fb620071f20 2024-05-27T06:58:29.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb62015aa70 con 0x7fb620071f20 2024-05-27T06:58:29.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6201f5ff0 con 0x7fb620071f20 2024-05-27T06:58:29.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.721+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb6100ad9f0 con 0x7fb620071f20 2024-05-27T06:58:29.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.725+0000 7fb61cc00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 0x7fb600071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:29.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.725+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb61012a4f0 con 0x7fb620071f20 2024-05-27T06:58:29.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.725+0000 7fb61fe00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 0x7fb600071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:29.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.725+0000 7fb61fe00640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 0x7fb600071940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb62015bc90 tx=0x7fb60c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:29.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.729+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb610016560 con 0x7fb620071f20 2024-05-27T06:58:29.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.917+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb620050870 con 0x7fb620071f20 2024-05-27T06:58:29.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb61cc00640 1 -- 172.21.15.81:0/3921754380 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fb6100f7150 con 0x7fb620071f20 2024-05-27T06:58:29.912 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:29.912 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:29.912 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:29.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 msgr2=0x7fb600071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 0x7fb600071940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb62015bc90 tx=0x7fb60c008040 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 msgr2=0x7fb62014b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb610019710 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 shutdown_connections 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb620160040 0x7fb6200cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb60006f430 0x7fb600071940 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb6200c9850 0x7fb62015bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 --2- 172.21.15.81:0/3921754380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb620071f20 0x7fb62014b730 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 >> 172.21.15.81:0/3921754380 conn(0x7fb620068d90 msgr2=0x7fb620160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 shutdown_connections 2024-05-27T06:58:29.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:29.921+0000 7fb626000640 1 -- 172.21.15.81:0/3921754380 wait complete. 2024-05-27T06:58:29.935 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:58:31.936 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:32.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~148s 2024-05-27T06:58:32.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 Processor -- start 2024-05-27T06:58:32.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 -- start start 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f433c0ca2f0 0x7f433c1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f433c0caca0 0x7f433c0cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f433c160c20 con 0x7f433c0d02a0 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f433c06eb20 con 0x7f433c0ca2f0 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4340a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f433c06f910 con 0x7f433c0caca0 2024-05-27T06:58:32.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f433c0ca2f0 0x7f433c1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f433c0ca2f0 0x7f433c1606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45700/0 (socket says 172.21.15.81:45700) 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 -- 172.21.15.81:0/910858082 learned_addr learned my addr 172.21.15.81:0/910858082 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 -- 172.21.15.81:0/910858082 >> v2:172.21.15.106:3300/0 conn(0x7f433c0caca0 msgr2=0x7f433c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 --2- 172.21.15.81:0/910858082 >> v2:172.21.15.106:3300/0 conn(0x7f433c0caca0 0x7f433c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 -- 172.21.15.81:0/910858082 >> v1:172.21.15.106:6789/0 conn(0x7f433c0d02a0 legacy=0x7f433c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:32.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.105+0000 7f4339e00640 1 -- 172.21.15.81:0/910858082 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433c15c710 con 0x7f433c0ca2f0 2024-05-27T06:58:32.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4339e00640 1 --2- 172.21.15.81:0/910858082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0ca2f0 0x7f433c1606e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f433c072fd0 tx=0x7f432c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d96bbc68fb0170 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:32.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4339400640 1 -- 172.21.15.81:0/910858082 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0b0070 con 0x7f433c0ca2f0 2024-05-27T06:58:32.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4339400640 1 -- 172.21.15.81:0/910858082 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432c0ab600 con 0x7f433c0ca2f0 2024-05-27T06:58:32.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4339400640 1 -- 172.21.15.81:0/910858082 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0ab9d0 con 0x7f433c0ca2f0 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 -- 172.21.15.81:0/910858082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0ca2f0 msgr2=0x7f433c1606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 --2- 172.21.15.81:0/910858082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0ca2f0 0x7f433c1606e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f433c072fd0 tx=0x7f432c01c540 comp rx=0 tx=0).stop 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 -- 172.21.15.81:0/910858082 shutdown_connections 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 --2- 172.21.15.81:0/910858082 >> v2:172.21.15.106:3300/0 conn(0x7f433c0caca0 0x7f433c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 --2- 172.21.15.81:0/910858082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0ca2f0 0x7f433c1606e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 -- 172.21.15.81:0/910858082 >> 172.21.15.81:0/910858082 conn(0x7f433c068ae0 msgr2=0x7f433c068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 -- 172.21.15.81:0/910858082 shutdown_connections 2024-05-27T06:58:32.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.109+0000 7f4340a00640 1 -- 172.21.15.81:0/910858082 wait complete. 2024-05-27T06:58:32.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 Processor -- start 2024-05-27T06:58:32.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 -- start start 2024-05-27T06:58:32.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 0x7f433c15e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 0x7f433c14c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f433c15cf90 con 0x7f433c0d02a0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f433c15ccb0 con 0x7f433c0caca0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4340a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f433c06e220 con 0x7f433c0ca2f0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45708/0 (socket says 172.21.15.81:45708) 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 -- 172.21.15.81:0/4273337979 learned_addr learned my addr 172.21.15.81:0/4273337979 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4339e00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 0x7f433c15e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4331400640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 0x7f433c14c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 -- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 msgr2=0x7f433c15e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 0x7f433c15e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 -- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 msgr2=0x7f433c14c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 0x7f433c14c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f433a800640 1 -- 172.21.15.81:0/4273337979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433c15ec80 con 0x7f433c0d02a0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4331400640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 0x7f433c14c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.113+0000 7f4339e00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 0x7f433c15e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f433a800640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f433400b990 tx=0x7f433400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334019070 con 0x7f433c0d02a0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f433400cba0 con 0x7f433c0d02a0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433c1605d0 con 0x7f433c0d02a0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334014440 con 0x7f433c0d02a0 2024-05-27T06:58:32.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f433c15df40 con 0x7f433c0d02a0 2024-05-27T06:58:32.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433c1f7020 con 0x7f433c0d02a0 2024-05-27T06:58:32.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f433401c070 con 0x7f433c0d02a0 2024-05-27T06:58:32.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 0x7f4318071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:32.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f433408f2f0 con 0x7f433c0d02a0 2024-05-27T06:58:32.111 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.117+0000 7f4339e00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 0x7f4318071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:32.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.121+0000 7f4339e00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 0x7f4318071910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f432c0ad000 tx=0x7f432c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:32.115 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.121+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f433405bf50 con 0x7f433c0d02a0 2024-05-27T06:58:32.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f433c06a560 con 0x7f433c0d02a0 2024-05-27T06:58:32.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4332a00640 1 -- 172.21.15.81:0/4273337979 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f433405b850 con 0x7f433c0d02a0 2024-05-27T06:58:32.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:32.305 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:32.305 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 msgr2=0x7f4318071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 0x7f4318071910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f432c0ad000 tx=0x7f432c008040 comp rx=0 tx=0).stop 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 msgr2=0x7f433c14ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f433400b990 tx=0x7f433400be50 comp rx=0 tx=0).stop 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 shutdown_connections 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f431806f400 0x7f4318071910 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f433c0d02a0 0x7f433c14ca60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f433c0caca0 0x7f433c14c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 --2- 172.21.15.81:0/4273337979 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f433c0ca2f0 0x7f433c15e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:32.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 >> 172.21.15.81:0/4273337979 conn(0x7f433c068ae0 msgr2=0x7f433c15af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:32.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 shutdown_connections 2024-05-27T06:58:32.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:32.313+0000 7f4340a00640 1 -- 172.21.15.81:0/4273337979 wait complete. 2024-05-27T06:58:32.327 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:34.329 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:34.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 Processor -- start 2024-05-27T06:58:34.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 -- start start 2024-05-27T06:58:34.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4a580c5f70 0x7f4a58165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:34.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a58072840 con 0x7f4a580c5b60 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a580c5270 con 0x7f4a58072430 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a60000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f4a580c5040 con 0x7f4a580c5f70 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4a580c5f70 0x7f4a58165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5cc00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4a580c5b60 0x7f4a58161cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34578/0 (socket says 172.21.15.81:34578) 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5cc00640 1 -- 172.21.15.81:0/2945002976 learned_addr learned my addr 172.21.15.81:0/2945002976 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:34.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a57e00640 1 -- 172.21.15.81:0/2945002976 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1339011020 0 0) 0x7f4a58072840 con 0x7f4a580c5b60 2024-05-27T06:58:34.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a57e00640 1 -- 172.21.15.81:0/2945002976 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a40005740 con 0x7f4a580c5b60 2024-05-27T06:58:34.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5e000640 1 -- 172.21.15.81:0/2945002976 >> v1:172.21.15.106:6789/0 conn(0x7f4a58072430 legacy=0x7f4a58159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5e000640 1 -- 172.21.15.81:0/2945002976 >> v1:172.21.15.81:6789/0 conn(0x7f4a580c5b60 legacy=0x7f4a58161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.497+0000 7f4a5e000640 1 -- 172.21.15.81:0/2945002976 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a580c7810 con 0x7f4a580c5f70 2024-05-27T06:58:34.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a5e000640 1 --2- 172.21.15.81:0/2945002976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5f70 0x7f4a58165400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009d90 tx=0x7f4a5001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b6e8cf8801d682 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:34.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a57e00640 1 -- 172.21.15.81:0/2945002976 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500b2070 con 0x7f4a580c5f70 2024-05-27T06:58:34.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a57e00640 1 -- 172.21.15.81:0/2945002976 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a500a5730 con 0x7f4a580c5f70 2024-05-27T06:58:34.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a57e00640 1 -- 172.21.15.81:0/2945002976 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500a5b00 con 0x7f4a580c5f70 2024-05-27T06:58:34.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 -- 172.21.15.81:0/2945002976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5f70 msgr2=0x7f4a58165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 --2- 172.21.15.81:0/2945002976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5f70 0x7f4a58165400 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4a50009d90 tx=0x7f4a5001c970 comp rx=0 tx=0).stop 2024-05-27T06:58:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 -- 172.21.15.81:0/2945002976 shutdown_connections 2024-05-27T06:58:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 --2- 172.21.15.81:0/2945002976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5f70 0x7f4a58165400 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 -- 172.21.15.81:0/2945002976 >> 172.21.15.81:0/2945002976 conn(0x7f4a58070700 msgr2=0x7f4a580cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 -- 172.21.15.81:0/2945002976 shutdown_connections 2024-05-27T06:58:34.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.501+0000 7f4a60000640 1 -- 172.21.15.81:0/2945002976 wait complete. 2024-05-27T06:58:34.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 Processor -- start 2024-05-27T06:58:34.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 -- start start 2024-05-27T06:58:34.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a58072430 0x7f4a581f94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:34.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a580c5f70 0x7f4a58146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4a580c7dd0 con 0x7f4a580c5b60 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4a580c75c0 con 0x7f4a580c5f70 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a60000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4a580ce6f0 con 0x7f4a58072430 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.505+0000 7f4a5cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:34.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45740/0 (socket says 172.21.15.81:45740) 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 -- 172.21.15.81:0/2602174394 learned_addr learned my addr 172.21.15.81:0/2602174394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5d600640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a58072430 0x7f4a581f94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 -- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a58072430 msgr2=0x7f4a581f94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a58072430 0x7f4a581f94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5e000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a580c5f70 0x7f4a58146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 -- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a580c5f70 msgr2=0x7f4a58146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a580c5f70 0x7f4a58146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 -- 172.21.15.81:0/2602174394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a58151c80 con 0x7f4a580c5b60 2024-05-27T06:58:34.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a5cc00640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400c430 tx=0x7f4a4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44017070 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a44005cf0 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a58151f60 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a440072d0 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a58074540 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5806a560 con 0x7f4a580c5b60 2024-05-27T06:58:34.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.509+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4a4401a070 con 0x7f4a580c5b60 2024-05-27T06:58:34.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.513+0000 7f4a55600640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 0x7f4a38071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:34.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.513+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4a4408e860 con 0x7f4a580c5b60 2024-05-27T06:58:34.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.513+0000 7f4a5d600640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 0x7f4a38071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:34.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.513+0000 7f4a5d600640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 0x7f4a38071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c000f70 tx=0x7f4a4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:34.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.517+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4a4405ae70 con 0x7f4a580c5b60 2024-05-27T06:58:34.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a580c5940 con 0x7f4a580c5b60 2024-05-27T06:58:34.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a55600640 1 -- 172.21.15.81:0/2602174394 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4a4405f170 con 0x7f4a580c5b60 2024-05-27T06:58:34.701 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:34.702 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:34.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 msgr2=0x7f4a38071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 0x7f4a38071910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c000f70 tx=0x7f4a4c009040 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 msgr2=0x7f4a5814bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4a4400c430 tx=0x7f4a4400c8f0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 shutdown_connections 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4a3806f400 0x7f4a38071910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a580c5f70 0x7f4a58146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a580c5b60 0x7f4a5814bb20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 --2- 172.21.15.81:0/2602174394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4a58072430 0x7f4a581f94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.709+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 >> 172.21.15.81:0/2602174394 conn(0x7f4a58070700 msgr2=0x7f4a58159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.713+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 shutdown_connections 2024-05-27T06:58:34.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:34.713+0000 7f4a60000640 1 -- 172.21.15.81:0/2602174394 wait complete. 2024-05-27T06:58:34.724 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:34.913 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:58:34.917 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:58:36.724 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:36.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 Processor -- start 2024-05-27T06:58:36.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 -- start start 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff0301680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03006e3e0 con 0x7ff0300ca2f0 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03006e6c0 con 0x7ff0300caca0 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff037600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff03015ce70 con 0x7ff03015d300 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff034c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff0300ca2f0 0x7ff030160710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52160/0 (socket says 172.21.15.81:52160) 2024-05-27T06:58:36.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff034c00640 1 -- 172.21.15.81:0/817769395 learned_addr learned my addr 172.21.15.81:0/817769395 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff0301680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1116246174 0 0) 0x7ff03006e3e0 con 0x7ff0300ca2f0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff018003530 con 0x7ff0300ca2f0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3056993060 0 0) 0x7ff03006e6c0 con 0x7ff0300caca0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff03006e3e0 con 0x7ff0300caca0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 197851927 0 0) 0x7ff018003530 con 0x7ff0300ca2f0 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 msgr2=0x7ff0301680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:36.888 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff0301680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 >> v1:172.21.15.106:6789/0 conn(0x7ff0300caca0 legacy=0x7ff0300cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03006e9a0 con 0x7ff0300ca2f0 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7ff024001700 con 0x7ff0300ca2f0 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff0301680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 >> v1:172.21.15.81:6789/0 conn(0x7ff0300ca2f0 legacy=0x7ff030160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff018003c70 0x7ff0180040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff018004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff018004da0 0x7ff018005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff03015ce70 con 0x7ff018004da0 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff01800bbd0 con 0x7ff018003c70 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff01800bdd0 con 0x7ff03015d300 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.893+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff018004da0 0x7ff018005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff02fe00640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff018004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff034c00640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff018003c70 0x7ff0180040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 -- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 msgr2=0x7ff018004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff018004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 -- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff018003c70 msgr2=0x7ff0180040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff018003c70 0x7ff0180040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 -- 172.21.15.81:0/817769395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff018003530 con 0x7ff018004da0 2024-05-27T06:58:36.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff02fe00640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff03015d300 0x7ff018004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff035600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff018004da0 0x7ff018005270 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff030073000 tx=0x7ff02801cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a6fb006cb8e520 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028009de0 con 0x7ff018004da0 2024-05-27T06:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0280a5db0 con 0x7ff018004da0 2024-05-27T06:58:36.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.897+0000 7ff02f400640 1 -- 172.21.15.81:0/817769395 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0280ad5f0 con 0x7ff018004da0 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- 172.21.15.81:0/817769395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff018004da0 msgr2=0x7ff018005270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 --2- 172.21.15.81:0/817769395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff018004da0 0x7ff018005270 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff030073000 tx=0x7ff02801cda0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff035600640 1 -- 172.21.15.81:0/817769395 reap_dead start 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- 172.21.15.81:0/817769395 shutdown_connections 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- 172.21.15.81:0/817769395 >> 172.21.15.81:0/817769395 conn(0x7ff030068ae0 msgr2=0x7ff030068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- 172.21.15.81:0/817769395 shutdown_connections 2024-05-27T06:58:36.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- 172.21.15.81:0/817769395 wait complete. 2024-05-27T06:58:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 Processor -- start 2024-05-27T06:58:36.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.901+0000 7ff037600640 1 -- start start 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff018004da0 0x7ff03014a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff030159bb0 0x7ff03014d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff03015d710 con 0x7ff0301f3a00 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff0300cb0b0 con 0x7ff030159bb0 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff03015cb00 con 0x7ff018004da0 2024-05-27T06:58:36.896 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47368/0 (socket says 172.21.15.81:47368) 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 -- 172.21.15.81:0/3618510660 learned_addr learned my addr 172.21.15.81:0/3618510660 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff034c00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff018004da0 0x7ff03014a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff035600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff030159bb0 0x7ff03014d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 -- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff018004da0 msgr2=0x7ff03014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff018004da0 0x7ff03014a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 -- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff030159bb0 msgr2=0x7ff03014d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff030159bb0 0x7ff03014d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 -- 172.21.15.81:0/3618510660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0301f7dd0 con 0x7ff0301f3a00 2024-05-27T06:58:36.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02fe00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff01c00e740 tx=0x7ff01c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c00ee50 con 0x7ff0301f3a00 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01c018070 con 0x7ff0301f3a00 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff030156170 con 0x7ff0301f3a00 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c00c6c0 con 0x7ff0301f3a00 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0301f72f0 con 0x7ff0301f3a00 2024-05-27T06:58:36.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff03006a560 con 0x7ff0301f3a00 2024-05-27T06:58:36.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.905+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff01c00c860 con 0x7ff0301f3a00 2024-05-27T06:58:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.909+0000 7ff02cc00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 0x7ff010071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.909+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff01c08f390 con 0x7ff0301f3a00 2024-05-27T06:58:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.909+0000 7ff034c00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 0x7ff010071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:36.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.909+0000 7ff034c00640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 0x7ff010071910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff024000fe0 tx=0x7ff024001500 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:36.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:36.913+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff01c05b910 con 0x7ff0301f3a00 2024-05-27T06:58:37.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.101+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff030159450 con 0x7ff0301f3a00 2024-05-27T06:58:37.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.101+0000 7ff02cc00640 1 -- 172.21.15.81:0/3618510660 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff01c05fc10 con 0x7ff0301f3a00 2024-05-27T06:58:37.094 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:37.094 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:37.094 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 msgr2=0x7ff010071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 0x7ff010071910 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff024000fe0 tx=0x7ff024001500 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 msgr2=0x7ff030159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff01c00e740 tx=0x7ff01c00ec00 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 shutdown_connections 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff01006f400 0x7ff010071910 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff030159bb0 0x7ff03014d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0301f3a00 0x7ff030159670 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 --2- 172.21.15.81:0/3618510660 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff018004da0 0x7ff03014a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 >> 172.21.15.81:0/3618510660 conn(0x7ff030068ae0 msgr2=0x7ff03014f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 shutdown_connections 2024-05-27T06:58:37.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:37.105+0000 7ff037600640 1 -- 172.21.15.81:0/3618510660 wait complete. 2024-05-27T06:58:37.116 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:38.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~154s 2024-05-27T06:58:39.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:39.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 Processor -- start 2024-05-27T06:58:39.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 -- start start 2024-05-27T06:58:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f294c0c82e0 0x7f294c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f294c0c8d40 0x7f294c0cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c0c97d0 con 0x7f294c0cf5c0 2024-05-27T06:58:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f294c0c9580 con 0x7f294c0c82e0 2024-05-27T06:58:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2952e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f294c0735a0 con 0x7f294c0c8d40 2024-05-27T06:58:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f294be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f294c0c82e0 0x7f294c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2950e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f294c0cf5c0 0x7f294c073ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52174/0 (socket says 172.21.15.81:52174) 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.285+0000 7f2950e00640 1 -- 172.21.15.81:0/2369756410 learned_addr learned my addr 172.21.15.81:0/2369756410 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f2943400640 1 --2- 172.21.15.81:0/2369756410 >> v2:172.21.15.106:3300/0 conn(0x7f294c0c8d40 0x7f294c0cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294be00640 1 -- 172.21.15.81:0/2369756410 >> v2:172.21.15.106:3300/0 conn(0x7f294c0c8d40 msgr2=0x7f294c0cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294be00640 1 --2- 172.21.15.81:0/2369756410 >> v2:172.21.15.106:3300/0 conn(0x7f294c0c8d40 0x7f294c0cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294be00640 1 -- 172.21.15.81:0/2369756410 >> v1:172.21.15.81:6789/0 conn(0x7f294c0cf5c0 legacy=0x7f294c073ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294be00640 1 -- 172.21.15.81:0/2369756410 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c0c9ab0 con 0x7f294c0c82e0 2024-05-27T06:58:39.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294b400640 1 -- 172.21.15.81:0/2369756410 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 265705119 0 0) 0x7f294c0c97d0 con 0x7f294c0cf5c0 2024-05-27T06:58:39.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294be00640 1 --2- 172.21.15.81:0/2369756410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c82e0 0x7f294c15d2c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f294c0cc740 tx=0x7f293c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcfd1ca566fa10b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:39.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294b400640 1 -- 172.21.15.81:0/2369756410 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c0b0070 con 0x7f294c0c82e0 2024-05-27T06:58:39.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294b400640 1 -- 172.21.15.81:0/2369756410 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c0ab600 con 0x7f294c0c82e0 2024-05-27T06:58:39.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.289+0000 7f294b400640 1 -- 172.21.15.81:0/2369756410 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c0ab9d0 con 0x7f294c0c82e0 2024-05-27T06:58:39.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- 172.21.15.81:0/2369756410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c82e0 msgr2=0x7f294c15d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 --2- 172.21.15.81:0/2369756410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c82e0 0x7f294c15d2c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f294c0cc740 tx=0x7f293c01c3f0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- 172.21.15.81:0/2369756410 shutdown_connections 2024-05-27T06:58:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 --2- 172.21.15.81:0/2369756410 >> v2:172.21.15.106:3300/0 conn(0x7f294c0c8d40 0x7f294c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 --2- 172.21.15.81:0/2369756410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c82e0 0x7f294c15d2c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- 172.21.15.81:0/2369756410 >> 172.21.15.81:0/2369756410 conn(0x7f294c06dca0 msgr2=0x7f294c06e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- 172.21.15.81:0/2369756410 shutdown_connections 2024-05-27T06:58:39.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- 172.21.15.81:0/2369756410 wait complete. 2024-05-27T06:58:39.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 Processor -- start 2024-05-27T06:58:39.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 -- start start 2024-05-27T06:58:39.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 0x7f294c14a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.293+0000 7f2952e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f294c0cf5c0 0x7f294c148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f294c169ce0 con 0x7f294c0c8d40 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f294c169b60 con 0x7f294c0cf5c0 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f294c169e60 con 0x7f294c0c82e0 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47378/0 (socket says 172.21.15.81:47378) 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 -- 172.21.15.81:0/1314496456 learned_addr learned my addr 172.21.15.81:0/1314496456 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2950e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f294c0cf5c0 0x7f294c148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f294be00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 0x7f294c14a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 -- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 msgr2=0x7f294c14a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 0x7f294c14a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 -- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f294c0cf5c0 msgr2=0x7f294c148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f294c0cf5c0 0x7f294c148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 -- 172.21.15.81:0/1314496456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c152e30 con 0x7f294c0c8d40 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f294be00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 0x7f294c14a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2943400640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f293400d5f0 tx=0x7f293400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340125f0 con 0x7f294c0c8d40 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2934019070 con 0x7f294c0c8d40 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c14ab00 con 0x7f294c0c8d40 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340129b0 con 0x7f294c0c8d40 2024-05-27T06:58:39.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c149ae0 con 0x7f294c0c8d40 2024-05-27T06:58:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c06a560 con 0x7f294c0c8d40 2024-05-27T06:58:39.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.297+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2934012b50 con 0x7f294c0c8d40 2024-05-27T06:58:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.301+0000 7f2948c00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 0x7f2928071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.301+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f293408f7d0 con 0x7f294c0c8d40 2024-05-27T06:58:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.301+0000 7f294be00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 0x7f2928071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:39.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.301+0000 7f294be00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 0x7f2928071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f293c0ad000 tx=0x7f293c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:39.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.305+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f294c152e30 con 0x7f294c0c8d40 2024-05-27T06:58:39.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.493+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f294c06d110 con 0x7f294c0c8d40 2024-05-27T06:58:39.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.493+0000 7f2948c00640 1 -- 172.21.15.81:0/1314496456 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f293405c4d0 con 0x7f294c0c8d40 2024-05-27T06:58:39.486 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:39.486 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:39.486 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:39.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 msgr2=0x7f2928071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 0x7f2928071940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f293c0ad000 tx=0x7f293c008040 comp rx=0 tx=0).stop 2024-05-27T06:58:39.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 msgr2=0x7f294c148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:39.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f293400d5f0 tx=0x7f293400dab0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 shutdown_connections 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f292806f430 0x7f2928071940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f294c0cf5c0 0x7f294c148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f294c0c8d40 0x7f294c148450 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 --2- 172.21.15.81:0/1314496456 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f294c0c82e0 0x7f294c14a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 >> 172.21.15.81:0/1314496456 conn(0x7f294c06dca0 msgr2=0x7f294c06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 shutdown_connections 2024-05-27T06:58:39.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:39.497+0000 7f2952e00640 1 -- 172.21.15.81:0/1314496456 wait complete. 2024-05-27T06:58:39.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:41.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 Processor -- start 2024-05-27T06:58:41.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 -- start start 2024-05-27T06:58:41.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fa5fc0caca0 0x7fa5fc0cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5fc160c20 con 0x7fa5fc0d02a0 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fa5fc06eb20 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa602400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fa5fc06f910 con 0x7fa5fc0caca0 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa5fb400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa5fb400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47388/0 (socket says 172.21.15.81:47388) 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.677+0000 7fa5fb400640 1 -- 172.21.15.81:0/3410332916 learned_addr learned my addr 172.21.15.81:0/3410332916 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5faa00640 1 --2- 172.21.15.81:0/3410332916 >> v2:172.21.15.106:3300/0 conn(0x7fa5fc0caca0 0x7fa5fc0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fb400640 1 -- 172.21.15.81:0/3410332916 >> v2:172.21.15.106:3300/0 conn(0x7fa5fc0caca0 msgr2=0x7fa5fc0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fb400640 1 --2- 172.21.15.81:0/3410332916 >> v2:172.21.15.106:3300/0 conn(0x7fa5fc0caca0 0x7fa5fc0cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fb400640 1 -- 172.21.15.81:0/3410332916 >> v1:172.21.15.106:6789/0 conn(0x7fa5fc0d02a0 legacy=0x7fa5fc0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fb400640 1 -- 172.21.15.81:0/3410332916 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc15c710 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fb400640 1 --2- 172.21.15.81:0/3410332916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc072fd0 tx=0x7fa5ec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7faf7a03b42b1c60 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:41.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fa000640 1 -- 172.21.15.81:0/3410332916 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0b0070 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fa000640 1 -- 172.21.15.81:0/3410332916 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec004230 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.681+0000 7fa5fa000640 1 -- 172.21.15.81:0/3410332916 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec003620 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- 172.21.15.81:0/3410332916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 msgr2=0x7fa5fc1606e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 --2- 172.21.15.81:0/3410332916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc072fd0 tx=0x7fa5ec01c520 comp rx=0 tx=0).stop 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- 172.21.15.81:0/3410332916 shutdown_connections 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 --2- 172.21.15.81:0/3410332916 >> v2:172.21.15.106:3300/0 conn(0x7fa5fc0caca0 0x7fa5fc0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 --2- 172.21.15.81:0/3410332916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc1606e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- 172.21.15.81:0/3410332916 >> 172.21.15.81:0/3410332916 conn(0x7fa5fc068ae0 msgr2=0x7fa5fc068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- 172.21.15.81:0/3410332916 shutdown_connections 2024-05-27T06:58:41.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- 172.21.15.81:0/3410332916 wait complete. 2024-05-27T06:58:41.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 Processor -- start 2024-05-27T06:58:41.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.685+0000 7fa602400640 1 -- start start 2024-05-27T06:58:41.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5fc0caca0 0x7fa5fc155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5fc0d02a0 0x7fa5fc15f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc15cf90 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa5fc15ccb0 con 0x7fa5fc0d02a0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fa5fc06e220 con 0x7fa5fc0caca0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47394/0 (socket says 172.21.15.81:47394) 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 -- 172.21.15.81:0/971580349 learned_addr learned my addr 172.21.15.81:0/971580349 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5faa00640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5fc0caca0 0x7fa5fc155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fbe00640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5fc0d02a0 0x7fa5fc15f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 -- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5fc0caca0 msgr2=0x7fa5fc155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5fc0caca0 0x7fa5fc155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 -- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5fc0d02a0 msgr2=0x7fa5fc15f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5fc0d02a0 0x7fa5fc15f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 -- 172.21.15.81:0/971580349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5fc151750 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5fb400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec01a150 tx=0x7fa5ec0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:41.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0b0040 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0bb070 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5fc150d30 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec003e60 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5fc1f6a50 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.689+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5fc06a560 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.693+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa5ec0c0460 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.693+0000 7fa5f3400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:41.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.693+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa5ec1297b0 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.693+0000 7fa5faa00640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:41.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.693+0000 7fa5faa00640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 0x7fa5dc071940 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc156350 tx=0x7fa5e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:41.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.697+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa5ec016110 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.889+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5fc14d2c0 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.889+0000 7fa5f3400640 1 -- 172.21.15.81:0/971580349 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fa5ec0f5dc0 con 0x7fa5fc0ca2f0 2024-05-27T06:58:41.882 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:41.882 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:41.882 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 msgr2=0x7fa5dc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 0x7fa5dc071940 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc156350 tx=0x7fa5e8008040 comp rx=0 tx=0).stop 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 msgr2=0x7fa5fc151210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec01a150 tx=0x7fa5ec0192c0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 shutdown_connections 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fa5fc0caca0 0x7fa5fc155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa5dc06f430 0x7fa5dc071940 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa5fc0d02a0 0x7fa5fc15f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 --2- 172.21.15.81:0/971580349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5fc0ca2f0 0x7fa5fc151210 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:41.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 >> 172.21.15.81:0/971580349 conn(0x7fa5fc068ae0 msgr2=0x7fa5fc14f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:41.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 shutdown_connections 2024-05-27T06:58:41.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:41.893+0000 7fa602400640 1 -- 172.21.15.81:0/971580349 wait complete. 2024-05-27T06:58:41.904 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:58:43.906 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:44.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.073+0000 7f2abee00640 1 Processor -- start 2024-05-27T06:58:44.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.073+0000 7f2abee00640 1 -- start start 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abee00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f2ab815ac20 0x7f2ab80c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abee00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab815db00 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abee00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab8073860 con 0x7f2ab81546f0 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abee00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab815e6f0 con 0x7f2ab815ac20 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2aaf400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2ab80c6fd0 0x7f2ab815a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52176/0 (socket says 172.21.15.81:52176) 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2aaf400640 1 -- 172.21.15.81:0/411850447 learned_addr learned my addr 172.21.15.81:0/411850447 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:44.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abce00640 1 --2- 172.21.15.81:0/411850447 >> v2:172.21.15.106:3300/0 conn(0x7f2ab815ac20 0x7f2ab80c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3302062094 0 0) 0x7f2ab815db00 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a9c003530 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abce00640 1 -- 172.21.15.81:0/411850447 >> v1:172.21.15.106:6789/0 conn(0x7f2ab81546f0 legacy=0x7f2ab8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abce00640 1 -- 172.21.15.81:0/411850447 >> v1:172.21.15.81:6789/0 conn(0x7f2ab80c6fd0 legacy=0x7f2ab815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1941491886 0 0) 0x7f2a9c003530 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.077+0000 7f2abce00640 1 -- 172.21.15.81:0/411850447 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab815ddc0 con 0x7f2ab815ac20 2024-05-27T06:58:44.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abce00640 1 --2- 172.21.15.81:0/411850447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab80c73e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3463bccaae712cb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:44.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00b5070 con 0x7f2ab815ac20 2024-05-27T06:58:44.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab00a8630 con 0x7f2ab815ac20 2024-05-27T06:58:44.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2ab7400640 1 -- 172.21.15.81:0/411850447 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0001700 con 0x7f2ab815ac20 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 -- 172.21.15.81:0/411850447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 msgr2=0x7f2ab80c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 --2- 172.21.15.81:0/411850447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab80c73e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001ebb0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 -- 172.21.15.81:0/411850447 shutdown_connections 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 --2- 172.21.15.81:0/411850447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab80c73e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 -- 172.21.15.81:0/411850447 >> 172.21.15.81:0/411850447 conn(0x7f2ab8070700 msgr2=0x7f2ab80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 -- 172.21.15.81:0/411850447 shutdown_connections 2024-05-27T06:58:44.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.081+0000 7f2abee00640 1 -- 172.21.15.81:0/411850447 wait complete. 2024-05-27T06:58:44.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 Processor -- start 2024-05-27T06:58:44.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- start start 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81546f0 0x7f2ab80ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab8155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab815e380 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80755b0 con 0x7f2ab815ac20 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab8154b00 con 0x7f2ab81546f0 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47402/0 (socket says 172.21.15.81:47402) 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 -- 172.21.15.81:0/3081182681 learned_addr learned my addr 172.21.15.81:0/3081182681 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abce00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab8155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:44.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2aaf400640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81546f0 0x7f2ab80ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 -- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81546f0 msgr2=0x7f2ab80ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81546f0 0x7f2ab80ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 -- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 msgr2=0x7f2ab8155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab8155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 -- 172.21.15.81:0/3081182681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab81f4810 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab7e00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c790 tx=0x7f2aa800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8003b80 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa8003d20 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab81552d0 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800aa50 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.085+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab8156fc0 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab8156a70 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f2aa800abf0 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2ab4c00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 0x7f2a94071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:44.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f2aa808f340 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2aaf400640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 0x7f2a94071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:44.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.089+0000 7f2aaf400640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 0x7f2a94071940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80cb2a0 tx=0x7f2aa0001c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:44.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.093+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f2aa805b950 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.285+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ab806a560 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.285+0000 7f2ab4c00640 1 -- 172.21.15.81:0/3081182681 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f2aa805fc50 con 0x7f2ab80c6fd0 2024-05-27T06:58:44.278 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:44.278 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:44.279 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 msgr2=0x7f2a94071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 0x7f2a94071940 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80cb2a0 tx=0x7f2aa0001c60 comp rx=0 tx=0).stop 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 msgr2=0x7f2ab81f42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800c790 tx=0x7f2aa800cc50 comp rx=0 tx=0).stop 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 shutdown_connections 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2ab81546f0 0x7f2ab80ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f2a9406f430 0x7f2a94071940 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab815ac20 0x7f2ab8155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 --2- 172.21.15.81:0/3081182681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ab80c6fd0 0x7f2ab81f42d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:44.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 >> 172.21.15.81:0/3081182681 conn(0x7f2ab8070700 msgr2=0x7f2ab80c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:44.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 shutdown_connections 2024-05-27T06:58:44.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:44.289+0000 7f2abee00640 1 -- 172.21.15.81:0/3081182681 wait complete. 2024-05-27T06:58:44.300 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:45.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~161s 2024-05-27T06:58:46.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 Processor -- start 2024-05-27T06:58:46.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 -- start start 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f29940c6de0 0x7f29940cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f29940ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29940755a0 con 0x7f29940cc930 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f29940c7400 con 0x7f29940c6de0 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f299bc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f29940c98c0 con 0x7f299415a3e0 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f2999c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f29940cc930 0x7f29940cf890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50016/0 (socket says 172.21.15.81:50016) 2024-05-27T06:58:46.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f2999c00640 1 -- 172.21.15.81:0/2777701680 learned_addr learned my addr 172.21.15.81:0/2777701680 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.469+0000 7f2999200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f29940c6de0 0x7f29940cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f298be00640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f29940ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2999200640 1 -- 172.21.15.81:0/2777701680 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 msgr2=0x7f29940ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2999200640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f29940ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2999200640 1 -- 172.21.15.81:0/2777701680 >> v1:172.21.15.81:6789/0 conn(0x7f29940cc930 legacy=0x7f29940cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2999200640 1 -- 172.21.15.81:0/2777701680 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940c9ba0 con 0x7f29940c6de0 2024-05-27T06:58:46.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2993e00640 1 -- 172.21.15.81:0/2777701680 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 501615149 0 0) 0x7f29940755a0 con 0x7f29940cc930 2024-05-27T06:58:46.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2999200640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940c6de0 0x7f29940cbd40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f29840097c0 tx=0x7f298401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd803a6864f831df server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:46.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2993e00640 1 -- 172.21.15.81:0/2777701680 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29840b5070 con 0x7f29940c6de0 2024-05-27T06:58:46.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2993e00640 1 -- 172.21.15.81:0/2777701680 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29840a89b0 con 0x7f29940c6de0 2024-05-27T06:58:46.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.473+0000 7f2993e00640 1 -- 172.21.15.81:0/2777701680 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29840a8d80 con 0x7f29940c6de0 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- 172.21.15.81:0/2777701680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940c6de0 msgr2=0x7f29940cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940c6de0 0x7f29940cbd40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f29840097c0 tx=0x7f298401ecb0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- 172.21.15.81:0/2777701680 shutdown_connections 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f29940ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 --2- 172.21.15.81:0/2777701680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940c6de0 0x7f29940cbd40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- 172.21.15.81:0/2777701680 >> 172.21.15.81:0/2777701680 conn(0x7f2994068ab0 msgr2=0x7f2994068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- 172.21.15.81:0/2777701680 shutdown_connections 2024-05-27T06:58:46.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- 172.21.15.81:0/2777701680 wait complete. 2024-05-27T06:58:46.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 Processor -- start 2024-05-27T06:58:46.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.477+0000 7f299bc00640 1 -- start start 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29940c6de0 0x7f2994150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f2994150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f29940c95c0 con 0x7f29940cc930 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f299415a7f0 con 0x7f29940c6de0 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f29940ca820 con 0x7f299415a3e0 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47414/0 (socket says 172.21.15.81:47414) 2024-05-27T06:58:46.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 -- 172.21.15.81:0/896000487 learned_addr learned my addr 172.21.15.81:0/896000487 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2999c00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f2994150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2999200640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29940c6de0 0x7f2994150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 -- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 msgr2=0x7f2994150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f2994150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 -- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29940c6de0 msgr2=0x7f2994150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29940c6de0 0x7f2994150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 -- 172.21.15.81:0/896000487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2994150590 con 0x7f29940cc930 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f298be00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e870 tx=0x7f297c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c017070 con 0x7f29940cc930 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f297c00c940 con 0x7f29940cc930 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2994157640 con 0x7f29940cc930 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297c00ccf0 con 0x7f29940cc930 2024-05-27T06:58:46.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29941578e0 con 0x7f29940cc930 2024-05-27T06:58:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f299406a560 con 0x7f29940cc930 2024-05-27T06:58:46.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.481+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f297c01a070 con 0x7f29940cc930 2024-05-27T06:58:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.485+0000 7f2991600640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 0x7f2970071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.485+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f297c08fcf0 con 0x7f29940cc930 2024-05-27T06:58:46.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.485+0000 7f2999200640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 0x7f2970071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:46.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.485+0000 7f2999200640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 0x7f2970071910 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f29840b2000 tx=0x7f2984008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:46.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.489+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f297c05c940 con 0x7f29940cc930 2024-05-27T06:58:46.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29940c90c0 con 0x7f29940cc930 2024-05-27T06:58:46.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f2991600640 1 -- 172.21.15.81:0/896000487 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f297c05c240 con 0x7f29940cc930 2024-05-27T06:58:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:46.673 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:46.674 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 msgr2=0x7f2970071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 0x7f2970071910 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f29840b2000 tx=0x7f2984008040 comp rx=0 tx=0).stop 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 msgr2=0x7f299414ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f297c00e870 tx=0x7f297c00ec70 comp rx=0 tx=0).stop 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 shutdown_connections 2024-05-27T06:58:46.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f299415a3e0 0x7f2994150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f297006f400 0x7f2970071910 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29940cc930 0x7f299414ad00 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 --2- 172.21.15.81:0/896000487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29940c6de0 0x7f2994150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 >> 172.21.15.81:0/896000487 conn(0x7f2994068ab0 msgr2=0x7f299406dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 shutdown_connections 2024-05-27T06:58:46.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:46.681+0000 7f299bc00640 1 -- 172.21.15.81:0/896000487 wait complete. 2024-05-27T06:58:46.695 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:48.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:48.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.865+0000 7f4fd3400640 1 Processor -- start 2024-05-27T06:58:48.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.865+0000 7f4fd3400640 1 -- start start 2024-05-27T06:58:48.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.865+0000 7f4fd3400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4fcc1546f0 0x7f4fcc072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.865+0000 7f4fd3400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 0x7f4fcc15a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.865+0000 7f4fd3400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fcc15db00 con 0x7f4fcc0732f0 2024-05-27T06:58:48.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd3400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4fcc15e6f0 con 0x7f4fcc1546f0 2024-05-27T06:58:48.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd3400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4fcc0755b0 con 0x7f4fcc15a510 2024-05-27T06:58:48.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd1400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 0x7f4fcc15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd0a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4fcc1546f0 0x7f4fcc072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fc3e00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4fcc0732f0 0x7f4fcc0c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:36484/0 (socket says 172.21.15.81:36484) 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fc3e00640 1 -- 172.21.15.81:0/2390741385 learned_addr learned my addr 172.21.15.81:0/2390741385 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fcbe00640 1 -- 172.21.15.81:0/2390741385 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009334555 0 0) 0x7f4fcc15db00 con 0x7f4fcc0732f0 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fcbe00640 1 -- 172.21.15.81:0/2390741385 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fb0003530 con 0x7f4fcc0732f0 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd0a00640 1 -- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 msgr2=0x7f4fcc15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd0a00640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 0x7f4fcc15a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd0a00640 1 -- 172.21.15.81:0/2390741385 >> v1:172.21.15.106:6789/0 conn(0x7f4fcc0732f0 legacy=0x7f4fcc0c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd0a00640 1 -- 172.21.15.81:0/2390741385 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc15ae60 con 0x7f4fcc1546f0 2024-05-27T06:58:48.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.869+0000 7f4fd1400640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 0x7f4fcc15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:58:48.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd0a00640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc1546f0 0x7f4fcc072db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097c0 tx=0x7f4fbc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d72060cfc101f44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:48.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fcbe00640 1 -- 172.21.15.81:0/2390741385 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc0b5070 con 0x7f4fcc1546f0 2024-05-27T06:58:48.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fcbe00640 1 -- 172.21.15.81:0/2390741385 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fbc0a8990 con 0x7f4fcc1546f0 2024-05-27T06:58:48.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fcbe00640 1 -- 172.21.15.81:0/2390741385 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc0a8d60 con 0x7f4fcc1546f0 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc1546f0 msgr2=0x7f4fcc072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc1546f0 0x7f4fcc072db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0097c0 tx=0x7f4fbc01ec90 comp rx=0 tx=0).stop 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- 172.21.15.81:0/2390741385 shutdown_connections 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc15a510 0x7f4fcc15a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 --2- 172.21.15.81:0/2390741385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc1546f0 0x7f4fcc072db0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- 172.21.15.81:0/2390741385 >> 172.21.15.81:0/2390741385 conn(0x7f4fcc070700 msgr2=0x7f4fcc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- 172.21.15.81:0/2390741385 shutdown_connections 2024-05-27T06:58:48.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- 172.21.15.81:0/2390741385 wait complete. 2024-05-27T06:58:48.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 Processor -- start 2024-05-27T06:58:48.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.873+0000 7f4fd3400640 1 -- start start 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc0732f0 0x7f4fcc1f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc1546f0 0x7f4fcc14d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc15e0a0 con 0x7f4fcc15a510 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc15ddc0 con 0x7f4fcc0732f0 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4fcc15e380 con 0x7f4fcc1546f0 2024-05-27T06:58:48.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47426/0 (socket says 172.21.15.81:47426) 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 -- 172.21.15.81:0/448153472 learned_addr learned my addr 172.21.15.81:0/448153472 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd0a00640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc0732f0 0x7f4fcc1f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fc3e00640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc1546f0 0x7f4fcc14d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 -- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc1546f0 msgr2=0x7f4fcc14d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc1546f0 0x7f4fcc14d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 -- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc0732f0 msgr2=0x7f4fcc1f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc0732f0 0x7f4fcc1f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 -- 172.21.15.81:0/448153472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc14da90 con 0x7f4fcc15a510 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd1400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400efb0 tx=0x7f4fc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4018020 con 0x7f4fcc15a510 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fc4014480 con 0x7f4fcc15a510 2024-05-27T06:58:48.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc14b940 con 0x7f4fcc15a510 2024-05-27T06:58:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4014850 con 0x7f4fcc15a510 2024-05-27T06:58:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc14bec0 con 0x7f4fcc15a510 2024-05-27T06:58:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.877+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc1579c0 con 0x7f4fcc15a510 2024-05-27T06:58:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.881+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4fc401b020 con 0x7f4fcc15a510 2024-05-27T06:58:48.872 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.881+0000 7f4fc9600640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 0x7f4fa8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.881+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4fc4090070 con 0x7f4fcc15a510 2024-05-27T06:58:48.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.881+0000 7f4fd0a00640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 0x7f4fa8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:48.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.881+0000 7f4fd0a00640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 0x7f4fa8071910 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0b2000 tx=0x7f4fbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:48.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:48.885+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4fc4022030 con 0x7f4fcc15a510 2024-05-27T06:58:49.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fcc06a560 con 0x7f4fcc15a510 2024-05-27T06:58:49.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fc9600640 1 -- 172.21.15.81:0/448153472 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4fc4010040 con 0x7f4fcc15a510 2024-05-27T06:58:49.069 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:49.069 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:49.069 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 msgr2=0x7f4fa8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 0x7f4fa8071910 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0b2000 tx=0x7f4fbc008040 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 msgr2=0x7f4fcc1537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400efb0 tx=0x7f4fc400c490 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 shutdown_connections 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4fcc1546f0 0x7f4fcc14d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4fa806f400 0x7f4fa8071910 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fcc15a510 0x7f4fcc1537a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 --2- 172.21.15.81:0/448153472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fcc0732f0 0x7f4fcc1f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 >> 172.21.15.81:0/448153472 conn(0x7f4fcc070700 msgr2=0x7f4fcc0c98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 shutdown_connections 2024-05-27T06:58:49.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:49.077+0000 7f4fd3400640 1 -- 172.21.15.81:0/448153472 wait complete. 2024-05-27T06:58:49.090 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:51.094 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 Processor -- start 2024-05-27T06:58:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 -- start start 2024-05-27T06:58:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 0x7fed540caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fed540cfdc0 0x7fed540d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed5406e3e0 con 0x7fed5415d300 2024-05-27T06:58:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fed5415ce70 con 0x7fed540cfdc0 2024-05-27T06:58:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed5ae00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fed5415d710 con 0x7fed540ca2f0 2024-05-27T06:58:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed4b400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fed5415d300 0x7fed54160710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50028/0 (socket says 172.21.15.81:50028) 2024-05-27T06:58:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed4b400640 1 -- 172.21.15.81:0/1280074008 learned_addr learned my addr 172.21.15.81:0/1280074008 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed58e00640 1 --2- 172.21.15.81:0/1280074008 >> v2:172.21.15.81:3300/0 conn(0x7fed540cfdc0 0x7fed540d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed53e00640 1 --2- 172.21.15.81:0/1280074008 >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 0x7fed540caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed58e00640 1 -- 172.21.15.81:0/1280074008 >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 msgr2=0x7fed540caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed58e00640 1 --2- 172.21.15.81:0/1280074008 >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 0x7fed540caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed58e00640 1 -- 172.21.15.81:0/1280074008 >> v1:172.21.15.81:6789/0 conn(0x7fed5415d300 legacy=0x7fed54160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed58e00640 1 -- 172.21.15.81:0/1280074008 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5406e6c0 con 0x7fed540cfdc0 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed53400640 1 -- 172.21.15.81:0/1280074008 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814212753 0 0) 0x7fed5406e3e0 con 0x7fed5415d300 2024-05-27T06:58:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.261+0000 7fed53e00640 1 --2- 172.21.15.81:0/1280074008 >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 0x7fed540caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed58e00640 1 --2- 172.21.15.81:0/1280074008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540cfdc0 0x7fed540d01d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fed540720e0 tx=0x7fed4c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbfbbe93c94df2dc server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed53400640 1 -- 172.21.15.81:0/1280074008 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0b2070 con 0x7fed540cfdc0 2024-05-27T06:58:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed53400640 1 -- 172.21.15.81:0/1280074008 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed4c0a5870 con 0x7fed540cfdc0 2024-05-27T06:58:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed53400640 1 -- 172.21.15.81:0/1280074008 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c0a5c40 con 0x7fed540cfdc0 2024-05-27T06:58:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 -- 172.21.15.81:0/1280074008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540cfdc0 msgr2=0x7fed540d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 --2- 172.21.15.81:0/1280074008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540cfdc0 0x7fed540d01d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fed540720e0 tx=0x7fed4c01c8a0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 -- 172.21.15.81:0/1280074008 shutdown_connections 2024-05-27T06:58:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 --2- 172.21.15.81:0/1280074008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540cfdc0 0x7fed540d01d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 --2- 172.21.15.81:0/1280074008 >> v2:172.21.15.106:3300/0 conn(0x7fed540ca2f0 0x7fed540caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 -- 172.21.15.81:0/1280074008 >> 172.21.15.81:0/1280074008 conn(0x7fed54068ae0 msgr2=0x7fed54068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.265+0000 7fed5ae00640 1 -- 172.21.15.81:0/1280074008 shutdown_connections 2024-05-27T06:58:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 -- 172.21.15.81:0/1280074008 wait complete. 2024-05-27T06:58:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 Processor -- start 2024-05-27T06:58:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 -- start start 2024-05-27T06:58:51.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 0x7fed54154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed5415d300 0x7fed5414a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed5406f880 con 0x7fed540ca2f0 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fed5406f700 con 0x7fed5415d300 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed5ae00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fed541698b0 con 0x7fed540cfdc0 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed53e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed53e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47442/0 (socket says 172.21.15.81:47442) 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed53e00640 1 -- 172.21.15.81:0/3061148511 learned_addr learned my addr 172.21.15.81:0/3061148511 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed58e00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed5415d300 0x7fed5414a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.269+0000 7fed4b400640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 0x7fed54154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 -- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 msgr2=0x7fed54154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 0x7fed54154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 -- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed5415d300 msgr2=0x7fed5414a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed5415d300 0x7fed5414a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 -- 172.21.15.81:0/3061148511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5414d510 con 0x7fed540ca2f0 2024-05-27T06:58:51.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed4b400640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 0x7fed54154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:51.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed53e00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed4400d830 tx=0x7fed4400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:51.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed44018020 con 0x7fed540ca2f0 2024-05-27T06:58:51.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed440096c0 con 0x7fed540ca2f0 2024-05-27T06:58:51.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed44009ab0 con 0x7fed540ca2f0 2024-05-27T06:58:51.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed5414b000 con 0x7fed540ca2f0 2024-05-27T06:58:51.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed5414ca30 con 0x7fed540ca2f0 2024-05-27T06:58:51.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed5406a560 con 0x7fed540ca2f0 2024-05-27T06:58:51.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fed4401b020 con 0x7fed540ca2f0 2024-05-27T06:58:51.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.273+0000 7fed50c00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 0x7fed30071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:51.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.277+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fed4408fb70 con 0x7fed540ca2f0 2024-05-27T06:58:51.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.277+0000 7fed4b400640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 0x7fed30071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:51.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.277+0000 7fed4b400640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 0x7fed30071910 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fed54153ee0 tx=0x7fed3c007cd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:51.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.281+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fed44022020 con 0x7fed540ca2f0 2024-05-27T06:58:51.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed540d02d0 con 0x7fed540ca2f0 2024-05-27T06:58:51.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed50c00640 1 -- 172.21.15.81:0/3061148511 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fed4405c7d0 con 0x7fed540ca2f0 2024-05-27T06:58:51.464 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:51.464 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:51.464 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:51.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 msgr2=0x7fed30071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 0x7fed30071910 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fed54153ee0 tx=0x7fed3c007cd0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 msgr2=0x7fed541f9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:51.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fed4400d830 tx=0x7fed4400dcf0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 shutdown_connections 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fed540cfdc0 0x7fed54154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fed3006f400 0x7fed30071910 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed5415d300 0x7fed5414a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 --2- 172.21.15.81:0/3061148511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed540ca2f0 0x7fed541f9570 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 >> 172.21.15.81:0/3061148511 conn(0x7fed54068ae0 msgr2=0x7fed540cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 shutdown_connections 2024-05-27T06:58:51.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:51.473+0000 7fed5ae00640 1 -- 172.21.15.81:0/3061148511 wait complete. 2024-05-27T06:58:51.486 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:52.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~168s 2024-05-27T06:58:53.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:53.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.657+0000 7f7448e00640 1 Processor -- start 2024-05-27T06:58:53.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.657+0000 7f7448e00640 1 -- start start 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7448e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f744415d3a0 0x7f744406d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7448e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f74440746b0 0x7f7444074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7448e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74440ca7c0 con 0x7f744406e7e0 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7448e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f74440ca590 con 0x7f744415d3a0 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7448e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f74440cad80 con 0x7f74440746b0 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f744415d3a0 0x7f744406d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7442800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f74440746b0 0x7f7444074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f744415d3a0 0x7f744406d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47450/0 (socket says 172.21.15.81:47450) 2024-05-27T06:58:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 -- 172.21.15.81:0/1403536600 learned_addr learned my addr 172.21.15.81:0/1403536600 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:53.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 -- 172.21.15.81:0/1403536600 >> v2:172.21.15.106:3300/0 conn(0x7f74440746b0 msgr2=0x7f7444074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:53.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 --2- 172.21.15.81:0/1403536600 >> v2:172.21.15.106:3300/0 conn(0x7f74440746b0 0x7f7444074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 -- 172.21.15.81:0/1403536600 >> v1:172.21.15.106:6789/0 conn(0x7f744406e7e0 legacy=0x7f74440cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.661+0000 7f7441e00640 1 -- 172.21.15.81:0/1403536600 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74440caaa0 con 0x7f744415d3a0 2024-05-27T06:58:53.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7441e00640 1 --2- 172.21.15.81:0/1403536600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f744406d8a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f74440718f0 tx=0x7f743401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b9c2d69866bc6c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:53.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7440a00640 1 -- 172.21.15.81:0/1403536600 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340ab430 con 0x7f744415d3a0 2024-05-27T06:58:53.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7440a00640 1 -- 172.21.15.81:0/1403536600 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340b0070 con 0x7f744415d3a0 2024-05-27T06:58:53.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7440a00640 1 -- 172.21.15.81:0/1403536600 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340ab7f0 con 0x7f744415d3a0 2024-05-27T06:58:53.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- 172.21.15.81:0/1403536600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 msgr2=0x7f744406d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 --2- 172.21.15.81:0/1403536600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f744406d8a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f74440718f0 tx=0x7f743401c520 comp rx=0 tx=0).stop 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- 172.21.15.81:0/1403536600 shutdown_connections 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 --2- 172.21.15.81:0/1403536600 >> v2:172.21.15.106:3300/0 conn(0x7f74440746b0 0x7f7444074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 --2- 172.21.15.81:0/1403536600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f744406d8a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- 172.21.15.81:0/1403536600 >> 172.21.15.81:0/1403536600 conn(0x7f7444072aa0 msgr2=0x7f74440cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- 172.21.15.81:0/1403536600 shutdown_connections 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- 172.21.15.81:0/1403536600 wait complete. 2024-05-27T06:58:53.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 Processor -- start 2024-05-27T06:58:53.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.665+0000 7f7448e00640 1 -- start start 2024-05-27T06:58:53.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f744406e7e0 0x7f744414b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74440746b0 0x7f7444155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f744406ebf0 con 0x7f744415d3a0 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f744415d7b0 con 0x7f74440746b0 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f7444169950 con 0x7f744406e7e0 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47466/0 (socket says 172.21.15.81:47466) 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 -- 172.21.15.81:0/3363569758 learned_addr learned my addr 172.21.15.81:0/3363569758 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7441400640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74440746b0 0x7f7444155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7441e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f744406e7e0 0x7f744414b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 -- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f744406e7e0 msgr2=0x7f744414b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f744406e7e0 0x7f744414b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 -- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74440746b0 msgr2=0x7f7444155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74440746b0 0x7f7444155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 -- 172.21.15.81:0/3363569758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744414faa0 con 0x7f744415d3a0 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7442800640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f743c009cb0 tx=0x7f743c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:53.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c018020 con 0x7f744415d3a0 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c014480 con 0x7f744415d3a0 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c014870 con 0x7f744415d3a0 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744414f4e0 con 0x7f744415d3a0 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7444156550 con 0x7f744415d3a0 2024-05-27T06:58:53.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.669+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744414aa40 con 0x7f744415d3a0 2024-05-27T06:58:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.673+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f743c01b020 con 0x7f744415d3a0 2024-05-27T06:58:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.673+0000 7f743a000640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 0x7f7424071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:53.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.673+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f743c090090 con 0x7f744415d3a0 2024-05-27T06:58:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.673+0000 7f7441e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 0x7f7424071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:53.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.673+0000 7f7441e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 0x7f7424071910 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f74340af000 tx=0x7f74340a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:53.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.677+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f743c022030 con 0x7f744415d3a0 2024-05-27T06:58:53.861 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.869+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f744406a560 con 0x7f744415d3a0 2024-05-27T06:58:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.869+0000 7f743a000640 1 -- 172.21.15.81:0/3363569758 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f743c010040 con 0x7f744415d3a0 2024-05-27T06:58:53.862 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:53.863 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:53.863 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:53.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 msgr2=0x7f7424071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 0x7f7424071910 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f74340af000 tx=0x7f74340a7000 comp rx=0 tx=0).stop 2024-05-27T06:58:53.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 msgr2=0x7f74441493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f743c009cb0 tx=0x7f743c00c490 comp rx=0 tx=0).stop 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 shutdown_connections 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f742406f400 0x7f7424071910 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f744415d3a0 0x7f74441493d0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74440746b0 0x7f7444155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 --2- 172.21.15.81:0/3363569758 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f744406e7e0 0x7f744414b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 >> 172.21.15.81:0/3363569758 conn(0x7f7444072aa0 msgr2=0x7f74440ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 shutdown_connections 2024-05-27T06:58:53.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:53.873+0000 7f7448e00640 1 -- 172.21.15.81:0/3363569758 wait complete. 2024-05-27T06:58:53.885 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:58:55.887 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.053+0000 7fd9b3800640 1 Processor -- start 2024-05-27T06:58:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 -- start start 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd9ac072430 0x7fd9ac159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac0c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9ac072840 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd9ac0c5040 con 0x7fd9ac072430 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b3800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd9ac0c75c0 con 0x7fd9ac15a040 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b1800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd9ac0c60a0 0x7fd9ac15c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34662/0 (socket says 172.21.15.81:34662) 2024-05-27T06:58:56.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b1800640 1 -- 172.21.15.81:0/3979587524 learned_addr learned my addr 172.21.15.81:0/3979587524 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/3979587524 >> v2:172.21.15.81:3300/0 conn(0x7fd9ac072430 0x7fd9ac159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9a3e00640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac0c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9abe00640 1 -- 172.21.15.81:0/3979587524 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268256047 0 0) 0x7fd9ac072840 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9abe00640 1 -- 172.21.15.81:0/3979587524 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd990003530 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b0e00640 1 -- 172.21.15.81:0/3979587524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 msgr2=0x7fd9ac0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac0c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b0e00640 1 -- 172.21.15.81:0/3979587524 >> v1:172.21.15.81:6789/0 conn(0x7fd9ac0c60a0 legacy=0x7fd9ac15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.057+0000 7fd9b0e00640 1 -- 172.21.15.81:0/3979587524 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac0c5270 con 0x7fd9ac072430 2024-05-27T06:58:56.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac072430 0x7fd9ac159b00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0097c0 tx=0x7fd99c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173edb5ee26d5ff5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:56.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9abe00640 1 -- 172.21.15.81:0/3979587524 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0b0070 con 0x7fd9ac072430 2024-05-27T06:58:56.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9abe00640 1 -- 172.21.15.81:0/3979587524 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99c0a4e50 con 0x7fd9ac072430 2024-05-27T06:58:56.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9abe00640 1 -- 172.21.15.81:0/3979587524 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0ab820 con 0x7fd9ac072430 2024-05-27T06:58:56.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 -- 172.21.15.81:0/3979587524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac072430 msgr2=0x7fd9ac159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac072430 0x7fd9ac159b00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0097c0 tx=0x7fd99c01c4a0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 -- 172.21.15.81:0/3979587524 shutdown_connections 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac0c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 --2- 172.21.15.81:0/3979587524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac072430 0x7fd9ac159b00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 -- 172.21.15.81:0/3979587524 >> 172.21.15.81:0/3979587524 conn(0x7fd9ac070700 msgr2=0x7fd9ac0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 -- 172.21.15.81:0/3979587524 shutdown_connections 2024-05-27T06:58:56.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.061+0000 7fd9b3800640 1 -- 172.21.15.81:0/3979587524 wait complete. 2024-05-27T06:58:56.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 Processor -- start 2024-05-27T06:58:56.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 -- start start 2024-05-27T06:58:56.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 0x7fd9ac1f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0ce6f0 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac0c7dd0 con 0x7fd9ac072430 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b3800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd9ac0c5610 con 0x7fd9ac15a040 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32994/0 (socket says 172.21.15.81:32994) 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 -- 172.21.15.81:0/18336417 learned_addr learned my addr 172.21.15.81:0/18336417 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 0x7fd9ac1f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b1800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 -- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 msgr2=0x7fd9ac1f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac1f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 -- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 msgr2=0x7fd9ac1f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 0x7fd9ac1f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 -- 172.21.15.81:0/18336417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac1522f0 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 0x7fd9ac1f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a3e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e870 tx=0x7fd99400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:56.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.065+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994017070 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99400c940 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac1fa140 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99400cd30 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac1f9400 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9ac06a560 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.061 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd99401a070 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.062 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9a9600640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 0x7fd988071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:56.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd99408f450 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.069+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 0x7fd988071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:56.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.073+0000 7fd9b0e00640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 0x7fd988071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0ad000 tx=0x7fd99c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:56.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.073+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd99405c020 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.265+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9ac0c5940 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.265+0000 7fd9a9600640 1 -- 172.21.15.81:0/18336417 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd99405b920 con 0x7fd9ac0c60a0 2024-05-27T06:58:56.258 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:56.258 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:56.258 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 msgr2=0x7fd988071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 0x7fd988071910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0ad000 tx=0x7fd99c008040 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 msgr2=0x7fd9ac158600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd99400e870 tx=0x7fd99400ec70 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 shutdown_connections 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd9ac15a040 0x7fd9ac1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd98806f400 0x7fd988071910 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9ac0c60a0 0x7fd9ac158600 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 --2- 172.21.15.81:0/18336417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9ac072430 0x7fd9ac1f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:56.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 >> 172.21.15.81:0/18336417 conn(0x7fd9ac070700 msgr2=0x7fd9ac06d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:56.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 shutdown_connections 2024-05-27T06:58:56.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:56.269+0000 7fd9b3800640 1 -- 172.21.15.81:0/18336417 wait complete. 2024-05-27T06:58:56.280 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:58.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:58:58.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 Processor -- start 2024-05-27T06:58:58.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 -- start start 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fc05806e7e0 0x7fc058168140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0580ca7c0 con 0x7fc05815d3a0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0580caaa0 con 0x7fc05806d8a0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05fc00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fc0580ca590 con 0x7fc05806e7e0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05d200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc05815d3a0 0x7fc0580cbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34674/0 (socket says 172.21.15.81:34674) 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05d200640 1 -- 172.21.15.81:0/3804685560 learned_addr learned my addr 172.21.15.81:0/3804685560 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc05dc00640 1 --2- 172.21.15.81:0/3804685560 >> v2:172.21.15.106:3300/0 conn(0x7fc05806e7e0 0x7fc058168140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1317795799 0 0) 0x7fc0580ca7c0 con 0x7fc05815d3a0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.449+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc040003530 con 0x7fc05815d3a0 2024-05-27T06:58:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 46396279 0 0) 0x7fc0580caaa0 con 0x7fc05806d8a0 2024-05-27T06:58:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0580ca7c0 con 0x7fc05806d8a0 2024-05-27T06:58:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3170777396 0 0) 0x7fc040003530 con 0x7fc05815d3a0 2024-05-27T06:58:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 >> v2:172.21.15.106:3300/0 conn(0x7fc05806e7e0 msgr2=0x7fc058168140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 --2- 172.21.15.81:0/3804685560 >> v2:172.21.15.106:3300/0 conn(0x7fc05806e7e0 0x7fc058168140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 >> v1:172.21.15.106:6789/0 conn(0x7fc05806d8a0 legacy=0x7fc0580746b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05806fe50 con 0x7fc05815d3a0 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fc04c001680 con 0x7fc05815d3a0 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 >> v1:172.21.15.81:6789/0 conn(0x7fc05815d3a0 legacy=0x7fc0580cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc040003c90 0x7fc0400040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc040004810 0x7fc040004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 0x7fc04000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc04000c260 con 0x7fc0400052f0 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc04000c410 con 0x7fc040004810 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc04000c610 con 0x7fc040003c90 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 0x7fc04000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05d200640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc040003c90 0x7fc0400040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 -- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc040003c90 msgr2=0x7fc0400040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc040003c90 0x7fc0400040e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 -- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc040004810 msgr2=0x7fc040004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:58:58.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057e00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc040004810 0x7fc040004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc040004810 0x7fc040004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 -- 172.21.15.81:0/3804685560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc040003530 con 0x7fc0400052f0 2024-05-27T06:58:58.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc057e00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc040004810 0x7fc040004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:58:58.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 -- 172.21.15.81:0/3804685560 reap_dead start 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.453+0000 7fc05dc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 0x7fc04000b9d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc050009f90 tx=0x7fc05001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8071eb4d6c84947 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050009c10 con 0x7fc0400052f0 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0500a5c90 con 0x7fc0400052f0 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc057400640 1 -- 172.21.15.81:0/3804685560 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050003be0 con 0x7fc0400052f0 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 msgr2=0x7fc04000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 0x7fc04000b9d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc050009f90 tx=0x7fc05001cc80 comp rx=0 tx=0).stop 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- 172.21.15.81:0/3804685560 shutdown_connections 2024-05-27T06:58:58.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 --2- 172.21.15.81:0/3804685560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0400052f0 0x7fc04000b9d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- 172.21.15.81:0/3804685560 >> 172.21.15.81:0/3804685560 conn(0x7fc058072aa0 msgr2=0x7fc0580cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:58.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- 172.21.15.81:0/3804685560 shutdown_connections 2024-05-27T06:58:58.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- 172.21.15.81:0/3804685560 wait complete. 2024-05-27T06:58:58.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 Processor -- start 2024-05-27T06:58:58.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.457+0000 7fc05fc00640 1 -- start start 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0400052f0 0x7fc0581e2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc05814a790 0x7fc05815b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc058070410 con 0x7fc0581f6a90 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc0580cad80 con 0x7fc05814a790 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fc05815d7b0 con 0x7fc0400052f0 2024-05-27T06:58:58.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33008/0 (socket says 172.21.15.81:33008) 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 -- 172.21.15.81:0/172895185 learned_addr learned my addr 172.21.15.81:0/172895185 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05d200640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0400052f0 0x7fc0581e2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05dc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc05814a790 0x7fc05815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 -- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0400052f0 msgr2=0x7fc0581e2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0400052f0 0x7fc0581e2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 -- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc05814a790 msgr2=0x7fc05815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc05814a790 0x7fc05815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 -- 172.21.15.81:0/172895185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0580c99c0 con 0x7fc0581f6a90 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc057e00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc04400d7a0 tx=0x7fc04400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044016070 con 0x7fc0581f6a90 2024-05-27T06:58:58.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc04400de50 con 0x7fc0581f6a90 2024-05-27T06:58:58.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0581f5fb0 con 0x7fc0581f6a90 2024-05-27T06:58:58.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04400bce0 con 0x7fc0581f6a90 2024-05-27T06:58:58.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0581e3850 con 0x7fc0581f6a90 2024-05-27T06:58:58.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.461+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0581f5dc0 con 0x7fc0581f6a90 2024-05-27T06:58:58.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.465+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fc044019070 con 0x7fc0581f6a90 2024-05-27T06:58:58.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.465+0000 7fc054c00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 0x7fc038071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:58:58.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.465+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fc04408f5a0 con 0x7fc0581f6a90 2024-05-27T06:58:58.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.465+0000 7fc05d200640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 0x7fc038071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:58:58.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.465+0000 7fc05d200640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 0x7fc038071940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0017f0 tx=0x7fc04c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:58:58.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.469+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fc04405bb20 con 0x7fc0581f6a90 2024-05-27T06:58:58.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc05806f6d0 con 0x7fc0581f6a90 2024-05-27T06:58:58.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc054c00640 1 -- 172.21.15.81:0/172895185 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fc0440641f0 con 0x7fc0581f6a90 2024-05-27T06:58:58.653 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:58:58.654 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:58:58.654 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 msgr2=0x7fc038071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 0x7fc038071940 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0017f0 tx=0x7fc04c009040 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 msgr2=0x7fc0581e81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc04400d7a0 tx=0x7fc04400dc60 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 shutdown_connections 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fc03806f430 0x7fc038071940 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc05814a790 0x7fc05815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0581f6a90 0x7fc0581e81b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 --2- 172.21.15.81:0/172895185 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fc0400052f0 0x7fc0581e2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 >> 172.21.15.81:0/172895185 conn(0x7fc058072aa0 msgr2=0x7fc058152c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 shutdown_connections 2024-05-27T06:58:58.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:58:58.661+0000 7fc05fc00640 1 -- 172.21.15.81:0/172895185 wait complete. 2024-05-27T06:58:58.677 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:58:58.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~175s 2024-05-27T06:59:00.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:00.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 Processor -- start 2024-05-27T06:59:00.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 -- start start 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 0x7f47380c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4738161cd0 0x7f47381640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4738072840 con 0x7f4738072430 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f47380c5040 con 0x7f4738161cd0 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.841+0000 7f473ea00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f47380c75c0 con 0x7f47380c5b60 2024-05-27T06:59:00.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4738072430 0x7f4738159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34688/0 (socket says 172.21.15.81:34688) 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737e00640 1 -- 172.21.15.81:0/4213113271 learned_addr learned my addr 172.21.15.81:0/4213113271 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f472fe00640 1 --2- 172.21.15.81:0/4213113271 >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 0x7f47380c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 --2- 172.21.15.81:0/4213113271 >> v2:172.21.15.81:3300/0 conn(0x7f4738161cd0 0x7f47381640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737400640 1 -- 172.21.15.81:0/4213113271 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084891156 0 0) 0x7f4738072840 con 0x7f4738072430 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737400640 1 -- 172.21.15.81:0/4213113271 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f471c003530 con 0x7f4738072430 2024-05-27T06:59:00.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 -- 172.21.15.81:0/4213113271 >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 msgr2=0x7f47380c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:00.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 --2- 172.21.15.81:0/4213113271 >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 0x7f47380c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:00.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 -- 172.21.15.81:0/4213113271 >> v1:172.21.15.81:6789/0 conn(0x7f4738072430 legacy=0x7f4738159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:00.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 -- 172.21.15.81:0/4213113271 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47380c5270 con 0x7f4738161cd0 2024-05-27T06:59:00.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f472fe00640 1 --2- 172.21.15.81:0/4213113271 >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 0x7f47380c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:00.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f473ca00640 1 --2- 172.21.15.81:0/4213113271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4738161cd0 0x7f47381640f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f473806f550 tx=0x7f473001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560c65a6379af47e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737400640 1 -- 172.21.15.81:0/4213113271 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47300b2070 con 0x7f4738161cd0 2024-05-27T06:59:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737400640 1 -- 172.21.15.81:0/4213113271 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47300a57f0 con 0x7f4738161cd0 2024-05-27T06:59:00.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.845+0000 7f4737400640 1 -- 172.21.15.81:0/4213113271 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47300a5bc0 con 0x7f4738161cd0 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- 172.21.15.81:0/4213113271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4738161cd0 msgr2=0x7f47381640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 --2- 172.21.15.81:0/4213113271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4738161cd0 0x7f47381640f0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f473806f550 tx=0x7f473001c930 comp rx=0 tx=0).stop 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- 172.21.15.81:0/4213113271 shutdown_connections 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 --2- 172.21.15.81:0/4213113271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4738161cd0 0x7f47381640f0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 --2- 172.21.15.81:0/4213113271 >> v2:172.21.15.106:3300/0 conn(0x7f47380c5b60 0x7f47380c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- 172.21.15.81:0/4213113271 >> 172.21.15.81:0/4213113271 conn(0x7f4738070700 msgr2=0x7f47380cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- 172.21.15.81:0/4213113271 shutdown_connections 2024-05-27T06:59:00.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- 172.21.15.81:0/4213113271 wait complete. 2024-05-27T06:59:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 Processor -- start 2024-05-27T06:59:00.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.849+0000 7f473ea00640 1 -- start start 2024-05-27T06:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4738072430 0x7f4738158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4738161cd0 0x7f47381f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f47380ce6f0 con 0x7f47380c5b60 2024-05-27T06:59:00.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f47380c7dd0 con 0x7f4738161cd0 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f47380c5610 con 0x7f4738072430 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33030/0 (socket says 172.21.15.81:33030) 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 -- 172.21.15.81:0/3196400958 learned_addr learned my addr 172.21.15.81:0/3196400958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f4737e00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4738072430 0x7f4738158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ca00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4738161cd0 0x7f47381f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 -- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4738072430 msgr2=0x7f4738158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4738072430 0x7f4738158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 -- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4738161cd0 msgr2=0x7f47381f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4738161cd0 0x7f47381f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:00.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 -- 172.21.15.81:0/3196400958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4738158780 con 0x7f47380c5b60 2024-05-27T06:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f472fe00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f472000e9b0 tx=0x7f472000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720017070 con 0x7f47380c5b60 2024-05-27T06:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f472000c9a0 con 0x7f47380c5b60 2024-05-27T06:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472000cd90 con 0x7f47380c5b60 2024-05-27T06:59:00.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4738074260 con 0x7f47380c5b60 2024-05-27T06:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4738074820 con 0x7f47380c5b60 2024-05-27T06:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.853+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f473806a560 con 0x7f47380c5b60 2024-05-27T06:59:00.847 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.857+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f472001a070 con 0x7f47380c5b60 2024-05-27T06:59:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.857+0000 7f4734c00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 0x7f4714071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.857+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f472008f3e0 con 0x7f47380c5b60 2024-05-27T06:59:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.857+0000 7f4737e00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 0x7f4714071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:00.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.857+0000 7f4737e00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 0x7f4714071910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4728007a70 tx=0x7f47280079c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:00.853 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:00.861+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f472005ba00 con 0x7f47380c5b60 2024-05-27T06:59:01.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.053+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47380c5940 con 0x7f47380c5b60 2024-05-27T06:59:01.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.053+0000 7f4734c00640 1 -- 172.21.15.81:0/3196400958 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f472005fd00 con 0x7f47380c5b60 2024-05-27T06:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:01.045 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:01.046 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 msgr2=0x7f4714071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 0x7f4714071910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4728007a70 tx=0x7f47280079c0 comp rx=0 tx=0).stop 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 msgr2=0x7f4738149680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f472000e9b0 tx=0x7f472000ee70 comp rx=0 tx=0).stop 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 shutdown_connections 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f471406f400 0x7f4714071910 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:01.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4738161cd0 0x7f47381f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:01.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47380c5b60 0x7f4738149680 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:01.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 --2- 172.21.15.81:0/3196400958 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4738072430 0x7f4738158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:01.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 >> 172.21.15.81:0/3196400958 conn(0x7f4738070700 msgr2=0x7f47380c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:01.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 shutdown_connections 2024-05-27T06:59:01.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:01.057+0000 7f473ea00640 1 -- 172.21.15.81:0/3196400958 wait complete. 2024-05-27T06:59:01.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:03.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:03.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.237+0000 7ff566a00640 1 Processor -- start 2024-05-27T06:59:03.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.237+0000 7ff566a00640 1 -- start start 2024-05-27T06:59:03.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff566a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff560072430 0x7ff560159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff566a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff56015a040 0x7ff5600c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff566a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff560072840 con 0x7ff5600c60a0 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff566a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7ff5600c5040 con 0x7ff56015a040 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff566a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff5600c75c0 con 0x7ff560072430 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff56015a040 0x7ff5600c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff564a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff5600c60a0 0x7ff56015c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34700/0 (socket says 172.21.15.81:34700) 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff564a00640 1 -- 172.21.15.81:0/1849308227 learned_addr learned my addr 172.21.15.81:0/1849308227 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:03.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff55fe00640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff560072430 0x7ff560159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 -- 172.21.15.81:0/1849308227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff560072430 msgr2=0x7ff560159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff560072430 0x7ff560159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 -- 172.21.15.81:0/1849308227 >> v1:172.21.15.81:6789/0 conn(0x7ff5600c60a0 legacy=0x7ff56015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 -- 172.21.15.81:0/1849308227 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5600c5270 con 0x7ff56015a040 2024-05-27T06:59:03.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff557400640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff5600c5b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe8629983a2656f server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:03.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.241+0000 7ff55f400640 1 -- 172.21.15.81:0/1849308227 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b3070 con 0x7ff56015a040 2024-05-27T06:59:03.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff55f400640 1 -- 172.21.15.81:0/1849308227 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480aa050 con 0x7ff56015a040 2024-05-27T06:59:03.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff55f400640 1 -- 172.21.15.81:0/1849308227 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480ae860 con 0x7ff56015a040 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- 172.21.15.81:0/1849308227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 msgr2=0x7ff5600c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff5600c5b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801e6b0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- 172.21.15.81:0/1849308227 shutdown_connections 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff5600c5b60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 --2- 172.21.15.81:0/1849308227 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff560072430 0x7ff560159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- 172.21.15.81:0/1849308227 >> 172.21.15.81:0/1849308227 conn(0x7ff560070700 msgr2=0x7ff5600cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- 172.21.15.81:0/1849308227 shutdown_connections 2024-05-27T06:59:03.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- 172.21.15.81:0/1849308227 wait complete. 2024-05-27T06:59:03.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 Processor -- start 2024-05-27T06:59:03.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.245+0000 7ff566a00640 1 -- start start 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff560072430 0x7ff56013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5600c60a0 0x7ff56013f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600ce6f0 con 0x7ff56015a040 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5600c7dd0 con 0x7ff560072430 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff5600c5610 con 0x7ff5600c60a0 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33040/0 (socket says 172.21.15.81:33040) 2024-05-27T06:59:03.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 -- 172.21.15.81:0/2982000320 learned_addr learned my addr 172.21.15.81:0/2982000320 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:03.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff55fe00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff560072430 0x7ff56013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff557400640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5600c60a0 0x7ff56013f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 -- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5600c60a0 msgr2=0x7ff56013f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5600c60a0 0x7ff56013f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 -- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff560072430 msgr2=0x7ff56013cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff560072430 0x7ff56013cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 -- 172.21.15.81:0/2982000320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56013d2e0 con 0x7ff56015a040 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff564a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff55800c3f0 tx=0x7ff55800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558017070 con 0x7ff56015a040 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff558005cf0 con 0x7ff56015a040 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff560146350 con 0x7ff56015a040 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580072d0 con 0x7ff56015a040 2024-05-27T06:59:03.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff560146910 con 0x7ff56015a040 2024-05-27T06:59:03.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.249+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff560145a30 con 0x7ff56015a040 2024-05-27T06:59:03.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.253+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff55801a070 con 0x7ff56015a040 2024-05-27T06:59:03.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.253+0000 7ff55cc00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 0x7ff53c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:03.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.253+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff55808e7d0 con 0x7ff56015a040 2024-05-27T06:59:03.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.253+0000 7ff55fe00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 0x7ff53c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:03.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.253+0000 7ff55fe00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 0x7ff53c071910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff560070fe0 tx=0x7ff55000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:03.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.257+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff55805ade0 con 0x7ff56015a040 2024-05-27T06:59:03.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff56006a560 con 0x7ff56015a040 2024-05-27T06:59:03.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff55cc00640 1 -- 172.21.15.81:0/2982000320 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff55805f0e0 con 0x7ff56015a040 2024-05-27T06:59:03.441 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:03.441 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:03.441 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:03.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 msgr2=0x7ff53c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 0x7ff53c071910 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff560070fe0 tx=0x7ff55000b040 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 msgr2=0x7ff56013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff55800c3f0 tx=0x7ff55800c8b0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 shutdown_connections 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff5600c60a0 0x7ff56013f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff53c06f400 0x7ff53c071910 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff56015a040 0x7ff56013f6d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 --2- 172.21.15.81:0/2982000320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff560072430 0x7ff56013cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 >> 172.21.15.81:0/2982000320 conn(0x7ff560070700 msgr2=0x7ff5600c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 shutdown_connections 2024-05-27T06:59:03.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:03.449+0000 7ff566a00640 1 -- 172.21.15.81:0/2982000320 wait complete. 2024-05-27T06:59:03.463 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:04.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:59:04.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:59:05.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:05.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~181s 2024-05-27T06:59:05.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 Processor -- start 2024-05-27T06:59:05.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 -- start start 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb6380c96b0 0x7fb638156820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb638160370 0x7fb638160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb6380cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fb63815cb60 con 0x7fb638160370 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb63815ce40 con 0x7fb6380c96b0 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63f000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb63815d120 con 0x7fb6380cebf0 2024-05-27T06:59:05.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb638160370 0x7fb638160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb638160370 0x7fb638160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40296/0 (socket says 172.21.15.81:40296) 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 -- 172.21.15.81:0/975067504 learned_addr learned my addr 172.21.15.81:0/975067504 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63dc00640 1 --2- 172.21.15.81:0/975067504 >> v2:172.21.15.106:3300/0 conn(0x7fb6380c96b0 0x7fb638156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63e600640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb6380cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 -- 172.21.15.81:0/975067504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 msgr2=0x7fb6380cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb6380cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.627 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 -- 172.21.15.81:0/975067504 >> v2:172.21.15.106:3300/0 conn(0x7fb6380c96b0 msgr2=0x7fb638156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 --2- 172.21.15.81:0/975067504 >> v2:172.21.15.106:3300/0 conn(0x7fb6380c96b0 0x7fb638156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.633+0000 7fb63d200640 1 -- 172.21.15.81:0/975067504 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63815d400 con 0x7fb638160370 2024-05-27T06:59:05.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63d200640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb638160370 0x7fb638160780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb63806d820 tx=0x7fb62401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549cc2ab2cef23d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:05.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb637e00640 1 -- 172.21.15.81:0/975067504 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6240b5070 con 0x7fb638160370 2024-05-27T06:59:05.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb637e00640 1 -- 172.21.15.81:0/975067504 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6240a89d0 con 0x7fb638160370 2024-05-27T06:59:05.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb637e00640 1 -- 172.21.15.81:0/975067504 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6240a8da0 con 0x7fb638160370 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 -- 172.21.15.81:0/975067504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb638160370 msgr2=0x7fb638160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb638160370 0x7fb638160780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb63806d820 tx=0x7fb62401ed10 comp rx=0 tx=0).stop 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 -- 172.21.15.81:0/975067504 shutdown_connections 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb6380cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 --2- 172.21.15.81:0/975067504 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb638160370 0x7fb638160780 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 --2- 172.21.15.81:0/975067504 >> v2:172.21.15.106:3300/0 conn(0x7fb6380c96b0 0x7fb638156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 -- 172.21.15.81:0/975067504 >> 172.21.15.81:0/975067504 conn(0x7fb638071390 msgr2=0x7fb6380717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 -- 172.21.15.81:0/975067504 shutdown_connections 2024-05-27T06:59:05.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.637+0000 7fb63f000640 1 -- 172.21.15.81:0/975067504 wait complete. 2024-05-27T06:59:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 Processor -- start 2024-05-27T06:59:05.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 -- start start 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb63814cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb638160370 0x7fb63814d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb638156ee0 con 0x7fb6380c96b0 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb638156d60 con 0x7fb638160370 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63f000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb63815c720 con 0x7fb6380cebf0 2024-05-27T06:59:05.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40300/0 (socket says 172.21.15.81:40300) 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63dc00640 1 -- 172.21.15.81:0/451333002 learned_addr learned my addr 172.21.15.81:0/451333002 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63d200640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb63814cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.641+0000 7fb63e600640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb638160370 0x7fb63814d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 -- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 msgr2=0x7fb63814cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb63814cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.636 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 -- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb638160370 msgr2=0x7fb63814d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:05.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb638160370 0x7fb63814d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 -- 172.21.15.81:0/451333002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63806e280 con 0x7fb6380c96b0 2024-05-27T06:59:05.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63dc00640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00e8a0 tx=0x7fb62c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c017070 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb62c00ee90 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb63815e8b0 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00cb80 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb63806ced0 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb638050870 con 0x7fb6380c96b0 2024-05-27T06:59:05.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.645+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb62c01a070 con 0x7fb6380c96b0 2024-05-27T06:59:05.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.649+0000 7fb635600640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 0x7fb618071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:05.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.649+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb62c08f2e0 con 0x7fb6380c96b0 2024-05-27T06:59:05.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.649+0000 7fb63d200640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 0x7fb618071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:05.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.649+0000 7fb63d200640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 0x7fb618071940 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6240b2000 tx=0x7fb624009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:05.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.653+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb62c05b8f0 con 0x7fb6380c96b0 2024-05-27T06:59:05.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.845+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6380c80f0 con 0x7fb6380c96b0 2024-05-27T06:59:05.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.845+0000 7fb635600640 1 -- 172.21.15.81:0/451333002 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fb62c05fbf0 con 0x7fb6380c96b0 2024-05-27T06:59:05.838 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:05.838 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 msgr2=0x7fb618071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 0x7fb618071940 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6240b2000 tx=0x7fb624009040 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 msgr2=0x7fb63815b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00e8a0 tx=0x7fb62c00ed60 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 shutdown_connections 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb6380cebf0 0x7fb63814cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb61806f430 0x7fb618071940 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb638160370 0x7fb63814d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 --2- 172.21.15.81:0/451333002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6380c96b0 0x7fb63815b200 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 >> 172.21.15.81:0/451333002 conn(0x7fb638071390 msgr2=0x7fb6380c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 shutdown_connections 2024-05-27T06:59:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:05.849+0000 7fb63f000640 1 -- 172.21.15.81:0/451333002 wait complete. 2024-05-27T06:59:05.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:59:07.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:08.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.029+0000 7f4327000640 1 Processor -- start 2024-05-27T06:59:08.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.029+0000 7f4327000640 1 -- start start 2024-05-27T06:59:08.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4327000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4320072430 0x7f4320159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4327000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43200c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4327000640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4320072840 con 0x7f43200c60a0 2024-05-27T06:59:08.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4327000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f43200c5040 con 0x7f4320072430 2024-05-27T06:59:08.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4327000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f43200c75c0 con 0x7f432015a040 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4317400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43200c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4320072430 0x7f4320159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4320072430 0x7f4320159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:43538/0 (socket says 172.21.15.81:43538) 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 -- 172.21.15.81:0/3139465879 learned_addr learned my addr 172.21.15.81:0/3139465879 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4317400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43200c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:43436/0 (socket says 172.21.15.81:43436) 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f4325000640 1 --1- 172.21.15.81:0/3139465879 >> v1:172.21.15.106:6789/0 conn(0x7f43200c60a0 0x7f432015c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:52446/0 (socket says 172.21.15.81:52446) 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 -- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 msgr2=0x7f43200c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 --2- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43200c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 -- 172.21.15.81:0/3139465879 >> v1:172.21.15.106:6789/0 conn(0x7f43200c60a0 legacy=0x7f432015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.033+0000 7f431fe00640 1 -- 172.21.15.81:0/3139465879 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43200c5270 con 0x7f4320072430 2024-05-27T06:59:08.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f431fe00640 1 --2- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4320072430 0x7f4320159b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f431000bb10 tx=0x7f431001ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d89e9f7cc97a7fd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:08.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f431f400640 1 -- 172.21.15.81:0/3139465879 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100b4070 con 0x7f4320072430 2024-05-27T06:59:08.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f431f400640 1 -- 172.21.15.81:0/3139465879 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43100a7a20 con 0x7f4320072430 2024-05-27T06:59:08.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f431f400640 1 -- 172.21.15.81:0/3139465879 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100a7df0 con 0x7f4320072430 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 -- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4320072430 msgr2=0x7f4320159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 --2- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4320072430 0x7f4320159b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f431000bb10 tx=0x7f431001ec30 comp rx=0 tx=0).stop 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 -- 172.21.15.81:0/3139465879 shutdown_connections 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 --2- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43200c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 --2- 172.21.15.81:0/3139465879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4320072430 0x7f4320159b00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 -- 172.21.15.81:0/3139465879 >> 172.21.15.81:0/3139465879 conn(0x7f4320070700 msgr2=0x7f43200cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 -- 172.21.15.81:0/3139465879 shutdown_connections 2024-05-27T06:59:08.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.037+0000 7f4327000640 1 -- 172.21.15.81:0/3139465879 wait complete. 2024-05-27T06:59:08.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 Processor -- start 2024-05-27T06:59:08.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- start start 2024-05-27T06:59:08.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43200c60a0 0x7f432014a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43201420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43200ce6f0 con 0x7f4320072430 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f43200c7dd0 con 0x7f43200c60a0 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f43200c5610 con 0x7f432015a040 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40304/0 (socket says 172.21.15.81:40304) 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 -- 172.21.15.81:0/3721312023 learned_addr learned my addr 172.21.15.81:0/3721312023 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4317400640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43200c60a0 0x7f432014a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4325000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43201420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 -- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 msgr2=0x7f43201420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43201420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 -- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43200c60a0 msgr2=0x7f432014a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43200c60a0 0x7f432014a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 -- 172.21.15.81:0/3721312023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432014add0 con 0x7f4320072430 2024-05-27T06:59:08.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431fe00640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f431001d2e0 tx=0x7f431001b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100c0070 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43100af7a0 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100b4040 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432015cdb0 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43200cce10 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.041+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f432006a560 con 0x7f4320072430 2024-05-27T06:59:08.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.045+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f43100afb90 con 0x7f4320072430 2024-05-27T06:59:08.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.045+0000 7f431cc00640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 0x7f42fc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:08.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.045+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f431012bbc0 con 0x7f4320072430 2024-05-27T06:59:08.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.045+0000 7f4317400640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 0x7f42fc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:08.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.045+0000 7f4317400640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 0x7f42fc071910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4320147ba0 tx=0x7f430800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:08.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.049+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4310018820 con 0x7f4320072430 2024-05-27T06:59:08.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4320076e40 con 0x7f4320072430 2024-05-27T06:59:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f431cc00640 1 -- 172.21.15.81:0/3721312023 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f43100f81d0 con 0x7f4320072430 2024-05-27T06:59:08.233 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:08.233 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:08.233 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 msgr2=0x7f42fc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 0x7f42fc071910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4320147ba0 tx=0x7f430800b040 comp rx=0 tx=0).stop 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 msgr2=0x7f4320158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f431001d2e0 tx=0x7f431001b9d0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 shutdown_connections 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f432015a040 0x7f43201420f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f42fc06f400 0x7f42fc071910 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43200c60a0 0x7f432014a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 --2- 172.21.15.81:0/3721312023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4320072430 0x7f4320158040 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 >> 172.21.15.81:0/3721312023 conn(0x7f4320070700 msgr2=0x7f43200749f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 shutdown_connections 2024-05-27T06:59:08.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:08.241+0000 7f4327000640 1 -- 172.21.15.81:0/3721312023 wait complete. 2024-05-27T06:59:08.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:10.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:10.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 Processor -- start 2024-05-27T06:59:10.416 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 -- start start 2024-05-27T06:59:10.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fceec0ca2e0 0x7fceec15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec15cc80 0x7fceec166fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceec0755b0 con 0x7fceec0c9630 2024-05-27T06:59:10.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fceec1608c0 con 0x7fceec0ca2e0 2024-05-27T06:59:10.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef4000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fceec0ca900 con 0x7fceec15cc80 2024-05-27T06:59:10.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef1600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fceec0c9630 0x7fceec0cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37780/0 (socket says 172.21.15.81:37780) 2024-05-27T06:59:10.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef1600640 1 -- 172.21.15.81:0/3485567372 learned_addr learned my addr 172.21.15.81:0/3485567372 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef2000640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec15cc80 0x7fceec166fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef0c00640 1 --2- 172.21.15.81:0/3485567372 >> v2:172.21.15.106:3300/0 conn(0x7fceec0ca2e0 0x7fceec15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716867685 0 0) 0x7fcee0001120 con 0x7fceec0c9630 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced4003530 con 0x7fceec0c9630 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 527540617 0 0) 0x7fced4003530 con 0x7fceec0c9630 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec15cc80 msgr2=0x7fceec166fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:10.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec15cc80 0x7fceec166fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 >> v2:172.21.15.106:3300/0 conn(0x7fceec0ca2e0 msgr2=0x7fceec15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 --2- 172.21.15.81:0/3485567372 >> v2:172.21.15.106:3300/0 conn(0x7fceec0ca2e0 0x7fceec15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec15fcb0 con 0x7fceec0c9630 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fcee00018d0 con 0x7fceec0c9630 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 >> v1:172.21.15.81:6789/0 conn(0x7fceec0c9630 legacy=0x7fceec0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fced4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced4004810 0x7fced4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fced4005160 0x7fced400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fceec1608c0 con 0x7fced4003a10 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fceec0ca900 con 0x7fced4004810 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fced400c010 con 0x7fced4005160 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef2000640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec15cc80 0x7fceec166fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:10.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef0c00640 1 --2- 172.21.15.81:0/3485567372 >> v2:172.21.15.106:3300/0 conn(0x7fceec0ca2e0 0x7fceec15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef1600640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fced4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef2000640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fced4005160 0x7fced400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.425+0000 7fcef0c00640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced4004810 0x7fced4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 -- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fced4005160 msgr2=0x7fced400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fced4005160 0x7fced400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 -- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced4004810 msgr2=0x7fced4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced4004810 0x7fced4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 -- 172.21.15.81:0/3485567372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced4003530 con 0x7fced4003a10 2024-05-27T06:59:10.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef1600640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fced4003e80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c4c0 tx=0x7fcee001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c881a98450fba2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee000c0e0 con 0x7fced4003a10 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00a9070 con 0x7fced4003a10 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fceebe00640 1 -- 172.21.15.81:0/3485567372 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005d20 con 0x7fced4003a10 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef4000640 1 -- 172.21.15.81:0/3485567372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 msgr2=0x7fced4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef4000640 1 --2- 172.21.15.81:0/3485567372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fced4003e80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c4c0 tx=0x7fcee001cc20 comp rx=0 tx=0).stop 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef2000640 1 -- 172.21.15.81:0/3485567372 reap_dead start 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef4000640 1 -- 172.21.15.81:0/3485567372 shutdown_connections 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.429+0000 7fcef4000640 1 -- 172.21.15.81:0/3485567372 >> 172.21.15.81:0/3485567372 conn(0x7fceec068ab0 msgr2=0x7fceec068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- 172.21.15.81:0/3485567372 shutdown_connections 2024-05-27T06:59:10.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- 172.21.15.81:0/3485567372 wait complete. 2024-05-27T06:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 Processor -- start 2024-05-27T06:59:10.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- start start 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceec15d8a0 0x7fceec0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 0x7fceec14b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fceec0ccfb0 con 0x7fced4003a10 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fceec15d090 con 0x7fceec15d8a0 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef4000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fceec0cd130 con 0x7fceec14ad00 2024-05-27T06:59:10.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.433+0000 7fcef1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40322/0 (socket says 172.21.15.81:40322) 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 -- 172.21.15.81:0/1902110606 learned_addr learned my addr 172.21.15.81:0/1902110606 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef0c00640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceec15d8a0 0x7fceec0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef2000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 0x7fceec14b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 -- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 msgr2=0x7fceec14b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 0x7fceec14b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 -- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceec15d8a0 msgr2=0x7fceec0cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceec15d8a0 0x7fceec0cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 -- 172.21.15.81:0/1902110606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceec1fa080 con 0x7fced4003a10 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef2000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 0x7fceec14b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef1600640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcee001b320 tx=0x7fced4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0001120 con 0x7fced4003a10 2024-05-27T06:59:10.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00a58b0 con 0x7fced4003a10 2024-05-27T06:59:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceec1f9380 con 0x7fced4003a10 2024-05-27T06:59:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee00a5ca0 con 0x7fced4003a10 2024-05-27T06:59:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceec15f850 con 0x7fced4003a10 2024-05-27T06:59:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceec06a560 con 0x7fced4003a10 2024-05-27T06:59:10.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.437+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fcee00047f0 con 0x7fced4003a10 2024-05-27T06:59:10.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.441+0000 7fcee9600640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 0x7fcecc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:10.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.441+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fcee0128e40 con 0x7fced4003a10 2024-05-27T06:59:10.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.441+0000 7fcef0c00640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 0x7fcecc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:10.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.441+0000 7fcef0c00640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 0x7fcecc071910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fceec077200 tx=0x7fced8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:10.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.445+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fceec0755b0 con 0x7fced4003a10 2024-05-27T06:59:10.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.633+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fceec074a80 con 0x7fced4003a10 2024-05-27T06:59:10.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.633+0000 7fcee9600640 1 -- 172.21.15.81:0/1902110606 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fcee00f5aa0 con 0x7fced4003a10 2024-05-27T06:59:10.627 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:10.627 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:10.627 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 msgr2=0x7fcecc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 0x7fcecc071910 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fceec077200 tx=0x7fced8008040 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 msgr2=0x7fceec1f9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fcee001b320 tx=0x7fced4005840 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 shutdown_connections 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fceec14ad00 0x7fceec14b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fcecc06f400 0x7fcecc071910 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fceec15d8a0 0x7fceec0cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 --2- 172.21.15.81:0/1902110606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4003a10 0x7fceec1f9b40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:10.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 >> 172.21.15.81:0/1902110606 conn(0x7fceec068ab0 msgr2=0x7fceec14bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 shutdown_connections 2024-05-27T06:59:10.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:10.637+0000 7fcef4000640 1 -- 172.21.15.81:0/1902110606 wait complete. 2024-05-27T06:59:10.651 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:12.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~188s 2024-05-27T06:59:12.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:12.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 Processor -- start 2024-05-27T06:59:12.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 -- start start 2024-05-27T06:59:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 0x7f4cf0072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 0x7f4cf00c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf015db00 con 0x7f4cf00c7510 2024-05-27T06:59:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4cf015e6f0 con 0x7f4cf01546f0 2024-05-27T06:59:12.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf8200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf00755b0 con 0x7f4cf00732f0 2024-05-27T06:59:12.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf5800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 0x7f4cf0072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf4e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 0x7f4cf00c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf6200640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7f4cf00c7510 0x7f4cf015a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:52460/0 (socket says 172.21.15.81:52460) 2024-05-27T06:59:12.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf6200640 1 -- 172.21.15.81:0/4242072529 learned_addr learned my addr 172.21.15.81:0/4242072529 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:12.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf4e00640 1 -- 172.21.15.81:0/4242072529 >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 msgr2=0x7f4cf0072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:12.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf4e00640 1 --2- 172.21.15.81:0/4242072529 >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 0x7f4cf0072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:12.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf4e00640 1 -- 172.21.15.81:0/4242072529 >> v1:172.21.15.106:6789/0 conn(0x7f4cf00c7510 legacy=0x7f4cf015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:12.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf4e00640 1 -- 172.21.15.81:0/4242072529 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf0073860 con 0x7f4cf00732f0 2024-05-27T06:59:12.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.821+0000 7f4cf5800640 1 --2- 172.21.15.81:0/4242072529 >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 0x7f4cf0072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf4e00640 1 --2- 172.21.15.81:0/4242072529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 0x7f4cf00c6fd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0097c0 tx=0x7f4cdc01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdc9befd1913132 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cefe00640 1 -- 172.21.15.81:0/4242072529 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0b5070 con 0x7f4cf00732f0 2024-05-27T06:59:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cefe00640 1 -- 172.21.15.81:0/4242072529 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cdc0a8990 con 0x7f4cf00732f0 2024-05-27T06:59:12.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cefe00640 1 -- 172.21.15.81:0/4242072529 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0a8d60 con 0x7f4cf00732f0 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 -- 172.21.15.81:0/4242072529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 msgr2=0x7f4cf00c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 --2- 172.21.15.81:0/4242072529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 0x7f4cf00c6fd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0097c0 tx=0x7f4cdc01ec90 comp rx=0 tx=0).stop 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 -- 172.21.15.81:0/4242072529 shutdown_connections 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 --2- 172.21.15.81:0/4242072529 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf00732f0 0x7f4cf00c6fd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 --2- 172.21.15.81:0/4242072529 >> v2:172.21.15.106:3300/0 conn(0x7f4cf01546f0 0x7f4cf0072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.825+0000 7f4cf8200640 1 -- 172.21.15.81:0/4242072529 >> 172.21.15.81:0/4242072529 conn(0x7f4cf0070700 msgr2=0x7f4cf00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:12.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- 172.21.15.81:0/4242072529 shutdown_connections 2024-05-27T06:59:12.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- 172.21.15.81:0/4242072529 wait complete. 2024-05-27T06:59:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 Processor -- start 2024-05-27T06:59:12.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- start start 2024-05-27T06:59:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf00732f0 0x7f4cf0159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf01546f0 0x7f4cf015ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf015e380 con 0x7f4cf00c7510 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf015e0a0 con 0x7f4cf00732f0 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf8200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf0154b00 con 0x7f4cf01546f0 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.829+0000 7f4cf4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40334/0 (socket says 172.21.15.81:40334) 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 -- 172.21.15.81:0/58676016 learned_addr learned my addr 172.21.15.81:0/58676016 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf6200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf01546f0 0x7f4cf015ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf5800640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf00732f0 0x7f4cf0159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 -- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf01546f0 msgr2=0x7f4cf015ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf01546f0 0x7f4cf015ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 -- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf00732f0 msgr2=0x7f4cf0159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:12.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf00732f0 0x7f4cf0159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:12.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 -- 172.21.15.81:0/58676016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf015a290 con 0x7f4cf00c7510 2024-05-27T06:59:12.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf4e00640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0b3040 tx=0x7f4cdc01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:12.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0bf070 con 0x7f4cf00c7510 2024-05-27T06:59:12.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cdc0b0720 con 0x7f4cf00c7510 2024-05-27T06:59:12.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc0b5040 con 0x7f4cf00c7510 2024-05-27T06:59:12.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf01f7100 con 0x7f4cf00c7510 2024-05-27T06:59:12.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf01589a0 con 0x7f4cf00c7510 2024-05-27T06:59:12.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf006a560 con 0x7f4cf00c7510 2024-05-27T06:59:12.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.833+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4cdc0c2070 con 0x7f4cf00c7510 2024-05-27T06:59:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.837+0000 7f4ced600640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 0x7f4cd0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.837+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4cdc12bbc0 con 0x7f4cf00c7510 2024-05-27T06:59:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.837+0000 7f4cf5800640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 0x7f4cd0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:12.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.837+0000 7f4cf5800640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 0x7f4cd0071910 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009ad0 tx=0x7f4ce4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:12.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:12.841+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4cdc0187f0 con 0x7f4cf00c7510 2024-05-27T06:59:13.024 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cf00c8e20 con 0x7f4cf00c7510 2024-05-27T06:59:13.025 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4ced600640 1 -- 172.21.15.81:0/58676016 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4cdc0f8820 con 0x7f4cf00c7510 2024-05-27T06:59:13.025 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:13.025 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:13.025 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:13.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 msgr2=0x7f4cd0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:13.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 0x7f4cd0071910 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009ad0 tx=0x7f4ce4008040 comp rx=0 tx=0).stop 2024-05-27T06:59:13.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 msgr2=0x7f4cf01f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:13.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc0b3040 tx=0x7f4cdc01ba30 comp rx=0 tx=0).stop 2024-05-27T06:59:13.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 shutdown_connections 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4cf01546f0 0x7f4cf015ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4cd006f400 0x7f4cd0071910 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4cf00c7510 0x7f4cf01f6bc0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 --2- 172.21.15.81:0/58676016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4cf00732f0 0x7f4cf0159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 >> 172.21.15.81:0/58676016 conn(0x7f4cf0070700 msgr2=0x7f4cf0151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.033+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 shutdown_connections 2024-05-27T06:59:13.028 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:13.037+0000 7f4cf8200640 1 -- 172.21.15.81:0/58676016 wait complete. 2024-05-27T06:59:13.047 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:15.048 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:15.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 Processor -- start 2024-05-27T06:59:15.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 -- start start 2024-05-27T06:59:15.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6e300ca770 0x7f6e300cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e300c78b0 con 0x7f6e300cbd00 2024-05-27T06:59:15.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e300c6ee0 con 0x7f6e30154200 2024-05-27T06:59:15.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e36a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6e300c71c0 con 0x7f6e300ca770 2024-05-27T06:59:15.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e27400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6e300cbd00 0x7f6e300cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37786/0 (socket says 172.21.15.81:37786) 2024-05-27T06:59:15.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e27400640 1 -- 172.21.15.81:0/2693511147 learned_addr learned my addr 172.21.15.81:0/2693511147 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266301181 0 0) 0x7f6e300c78b0 con 0x7f6e300cbd00 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/2693511147 >> v2:172.21.15.106:3300/0 conn(0x7f6e300ca770 0x7f6e300cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.217+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e14003530 con 0x7f6e300cbd00 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3842011694 0 0) 0x7f6e14003530 con 0x7f6e300cbd00 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 >> v2:172.21.15.106:3300/0 conn(0x7f6e300ca770 msgr2=0x7f6e300cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 --2- 172.21.15.81:0/2693511147 >> v2:172.21.15.106:3300/0 conn(0x7f6e300ca770 0x7f6e300cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 >> v1:172.21.15.106:6789/0 conn(0x7f6e30154200 legacy=0x7f6e30164d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e300c7b70 con 0x7f6e300cbd00 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f6e18001700 con 0x7f6e300cbd00 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 >> v1:172.21.15.81:6789/0 conn(0x7f6e300cbd00 legacy=0x7f6e300cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 0x7f6e14003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14004810 0x7f6e14004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e14005260 0x7f6e1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e300c71c0 con 0x7f6e14003a60 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1400c1f0 con 0x7f6e14005260 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6e1400c3f0 con 0x7f6e14004810 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 0x7f6e14003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e34a00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e14005260 0x7f6e1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e27400640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14004810 0x7f6e14004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 -- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14004810 msgr2=0x7f6e14004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14004810 0x7f6e14004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 -- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e14005260 msgr2=0x7f6e1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e14005260 0x7f6e1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e34a00640 1 -- 172.21.15.81:0/2693511147 reap_dead start 2024-05-27T06:59:15.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.221+0000 7f6e2fe00640 1 -- 172.21.15.81:0/2693511147 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e14003530 con 0x7f6e14003a60 2024-05-27T06:59:15.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 0x7f6e14003ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009ae0 tx=0x7f6e2001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dcdc60829fdb14 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:15.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e200b2070 con 0x7f6e14003a60 2024-05-27T06:59:15.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e200a5550 con 0x7f6e14003a60 2024-05-27T06:59:15.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e2f400640 1 -- 172.21.15.81:0/2693511147 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e200096b0 con 0x7f6e14003a60 2024-05-27T06:59:15.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 msgr2=0x7f6e14003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 0x7f6e14003ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009ae0 tx=0x7f6e2001ca70 comp rx=0 tx=0).stop 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- 172.21.15.81:0/2693511147 shutdown_connections 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 --2- 172.21.15.81:0/2693511147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14003a60 0x7f6e14003ed0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- 172.21.15.81:0/2693511147 >> 172.21.15.81:0/2693511147 conn(0x7f6e30068ae0 msgr2=0x7f6e30068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- 172.21.15.81:0/2693511147 shutdown_connections 2024-05-27T06:59:15.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- 172.21.15.81:0/2693511147 wait complete. 2024-05-27T06:59:15.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 Processor -- start 2024-05-27T06:59:15.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.225+0000 7f6e36a00640 1 -- start start 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14003a60 0x7f6e301f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e30154e90 0x7f6e301f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e300cc2f0 con 0x7f6e301552a0 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6e30154a20 con 0x7f6e30154e90 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6e300cc470 con 0x7f6e14003a60 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40360/0 (socket says 172.21.15.81:40360) 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 -- 172.21.15.81:0/1617463139 learned_addr learned my addr 172.21.15.81:0/1617463139 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14003a60 0x7f6e301f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e27400640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e30154e90 0x7f6e301f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 -- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14003a60 msgr2=0x7f6e301f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14003a60 0x7f6e301f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 -- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e30154e90 msgr2=0x7f6e301f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e30154e90 0x7f6e301f74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 -- 172.21.15.81:0/1617463139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3014cef0 con 0x7f6e301552a0 2024-05-27T06:59:15.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e34a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6e300c88a0 tx=0x7f6e2800fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:15.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28003980 con 0x7f6e301552a0 2024-05-27T06:59:15.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e28018070 con 0x7f6e301552a0 2024-05-27T06:59:15.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28003d80 con 0x7f6e301552a0 2024-05-27T06:59:15.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e301f4d80 con 0x7f6e301552a0 2024-05-27T06:59:15.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e30157890 con 0x7f6e301552a0 2024-05-27T06:59:15.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.229+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e3014d0c0 con 0x7f6e301552a0 2024-05-27T06:59:15.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.233+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6e280052e0 con 0x7f6e301552a0 2024-05-27T06:59:15.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.233+0000 7f6e2cc00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 0x7f6e0c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:15.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.233+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6e2808f4f0 con 0x7f6e301552a0 2024-05-27T06:59:15.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.233+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 0x7f6e0c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:15.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.233+0000 7f6e2fe00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 0x7f6e0c071940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e200af000 tx=0x7f6e200046d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:15.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.237+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6e2805c140 con 0x7f6e301552a0 2024-05-27T06:59:15.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.429+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e3006a560 con 0x7f6e301552a0 2024-05-27T06:59:15.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.429+0000 7f6e2cc00640 1 -- 172.21.15.81:0/1617463139 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6e2805ba40 con 0x7f6e301552a0 2024-05-27T06:59:15.421 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:15.422 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:15.422 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 msgr2=0x7f6e0c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 0x7f6e0c071940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e200af000 tx=0x7f6e200046d0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 msgr2=0x7f6e301f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6e300c88a0 tx=0x7f6e2800fbd0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 shutdown_connections 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6e0c06f430 0x7f6e0c071940 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e301552a0 0x7f6e301f79e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6e30154e90 0x7f6e301f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 --2- 172.21.15.81:0/1617463139 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6e14003a60 0x7f6e301f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 >> 172.21.15.81:0/1617463139 conn(0x7f6e30068ae0 msgr2=0x7f6e30153630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 shutdown_connections 2024-05-27T06:59:15.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:15.433+0000 7f6e36a00640 1 -- 172.21.15.81:0/1617463139 wait complete. 2024-05-27T06:59:15.444 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:17.445 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:17.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 Processor -- start 2024-05-27T06:59:17.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 -- start start 2024-05-27T06:59:17.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 0x7f5b18154550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 0x7f5b180cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b18154a90 con 0x7f5b180cebf0 2024-05-27T06:59:17.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5b180c9170 con 0x7f5b1815e6b0 2024-05-27T06:59:17.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5b18074bf0 con 0x7f5b1815b050 2024-05-27T06:59:17.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b0fe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5b180cebf0 0x7f5b1815dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34498/0 (socket says 172.21.15.81:34498) 2024-05-27T06:59:17.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b0fe00640 1 -- 172.21.15.81:0/2751032435 learned_addr learned my addr 172.21.15.81:0/2751032435 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:17.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/2751032435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 0x7f5b18154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.613+0000 7f5b1d400640 1 --2- 172.21.15.81:0/2751032435 >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 0x7f5b180cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b17e00640 1 -- 172.21.15.81:0/2751032435 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517708060 0 0) 0x7f5b18154a90 con 0x7f5b180cebf0 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b17e00640 1 -- 172.21.15.81:0/2751032435 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5afc003530 con 0x7f5b180cebf0 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1ca00640 1 -- 172.21.15.81:0/2751032435 >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 msgr2=0x7f5b180cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/2751032435 >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 0x7f5b180cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1ca00640 1 -- 172.21.15.81:0/2751032435 >> v1:172.21.15.81:6789/0 conn(0x7f5b180cebf0 legacy=0x7f5b1815dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1ca00640 1 -- 172.21.15.81:0/2751032435 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b180c93a0 con 0x7f5b1815b050 2024-05-27T06:59:17.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1d400640 1 --2- 172.21.15.81:0/2751032435 >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 0x7f5b180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:17.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/2751032435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 0x7f5b18154550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b080097c0 tx=0x7f5b0801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da598683edb9de9e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:17.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b17e00640 1 -- 172.21.15.81:0/2751032435 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080b5070 con 0x7f5b1815b050 2024-05-27T06:59:17.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b17e00640 1 -- 172.21.15.81:0/2751032435 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b080a8990 con 0x7f5b1815b050 2024-05-27T06:59:17.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.617+0000 7f5b17e00640 1 -- 172.21.15.81:0/2751032435 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080a8d60 con 0x7f5b1815b050 2024-05-27T06:59:17.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- 172.21.15.81:0/2751032435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 msgr2=0x7f5b18154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 --2- 172.21.15.81:0/2751032435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 0x7f5b18154550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b080097c0 tx=0x7f5b0801ec90 comp rx=0 tx=0).stop 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- 172.21.15.81:0/2751032435 shutdown_connections 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 --2- 172.21.15.81:0/2751032435 >> v2:172.21.15.106:3300/0 conn(0x7f5b1815e6b0 0x7f5b180cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 --2- 172.21.15.81:0/2751032435 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815b050 0x7f5b18154550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- 172.21.15.81:0/2751032435 >> 172.21.15.81:0/2751032435 conn(0x7f5b18071390 msgr2=0x7f5b180717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- 172.21.15.81:0/2751032435 shutdown_connections 2024-05-27T06:59:17.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- 172.21.15.81:0/2751032435 wait complete. 2024-05-27T06:59:17.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 Processor -- start 2024-05-27T06:59:17.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.621+0000 7f5b1de00640 1 -- start start 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 0x7f5b181f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815e6b0 0x7f5b1806da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b18075100 con 0x7f5b180cebf0 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5b18074e20 con 0x7f5b1815b050 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1815b460 con 0x7f5b1815e6b0 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49764/0 (socket says 172.21.15.81:49764) 2024-05-27T06:59:17.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 -- 172.21.15.81:0/1897365952 learned_addr learned my addr 172.21.15.81:0/1897365952 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:17.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 -- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815e6b0 msgr2=0x7f5b1806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.617 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815e6b0 0x7f5b1806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b0fe00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 0x7f5b181f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 -- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 msgr2=0x7f5b181f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 0x7f5b181f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 -- 172.21.15.81:0/1897365952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b181f5f20 con 0x7f5b180cebf0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b0fe00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 0x7f5b181f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1ca00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5b0801c8c0 tx=0x7f5b0801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080bf070 con 0x7f5b180cebf0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b080b0660 con 0x7f5b180cebf0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b080b5040 con 0x7f5b180cebf0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b181f6200 con 0x7f5b180cebf0 2024-05-27T06:59:17.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b181f2e40 con 0x7f5b180cebf0 2024-05-27T06:59:17.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.625+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b181f30a0 con 0x7f5b180cebf0 2024-05-27T06:59:17.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.629+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5b080c2070 con 0x7f5b180cebf0 2024-05-27T06:59:17.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.629+0000 7f5b15600640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 0x7f5af4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:17.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.629+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f5b0812ba00 con 0x7f5b180cebf0 2024-05-27T06:59:17.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.629+0000 7f5b0fe00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 0x7f5af4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:17.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.629+0000 7f5b0fe00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 0x7f5af4071940 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5b18153f50 tx=0x7f5b00009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:17.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.633+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5b080187f0 con 0x7f5b180cebf0 2024-05-27T06:59:17.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.825+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b18050870 con 0x7f5b180cebf0 2024-05-27T06:59:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.825+0000 7f5b15600640 1 -- 172.21.15.81:0/1897365952 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f5b080f8700 con 0x7f5b180cebf0 2024-05-27T06:59:17.819 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:17.819 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:17.819 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 msgr2=0x7f5af4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 0x7f5af4071940 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5b18153f50 tx=0x7f5b00009040 comp rx=0 tx=0).stop 2024-05-27T06:59:17.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 msgr2=0x7f5b180c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5b0801c8c0 tx=0x7f5b0801ba30 comp rx=0 tx=0).stop 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 shutdown_connections 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5b1815e6b0 0x7f5b1806da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5af406f430 0x7f5af4071940 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5b1815b050 0x7f5b181f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 --2- 172.21.15.81:0/1897365952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b180cebf0 0x7f5b180c5c30 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 >> 172.21.15.81:0/1897365952 conn(0x7f5b18071390 msgr2=0x7f5b18073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 shutdown_connections 2024-05-27T06:59:17.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:17.829+0000 7f5b1de00640 1 -- 172.21.15.81:0/1897365952 wait complete. 2024-05-27T06:59:17.841 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:59:19.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~195s 2024-05-27T06:59:19.843 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:20.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 Processor -- start 2024-05-27T06:59:20.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 -- start start 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49c4072430 0x7f49c4159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f49c415a040 0x7f49c40c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c40c60a0 0x7f49c415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f49c40c5040 con 0x7f49c4072430 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f49c40c75c0 con 0x7f49c415a040 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49cc000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f49c40c7dd0 con 0x7f49c40c60a0 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49c4072430 0x7f49c4159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49c4072430 0x7f49c4159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49774/0 (socket says 172.21.15.81:49774) 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 -- 172.21.15.81:0/1301301534 learned_addr learned my addr 172.21.15.81:0/1301301534 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c8c00640 1 --2- 172.21.15.81:0/1301301534 >> v2:172.21.15.106:3300/0 conn(0x7f49c415a040 0x7f49c40c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49ca000640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c40c60a0 0x7f49c415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 -- 172.21.15.81:0/1301301534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c40c60a0 msgr2=0x7f49c415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c40c60a0 0x7f49c415c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 -- 172.21.15.81:0/1301301534 >> v2:172.21.15.106:3300/0 conn(0x7f49c415a040 msgr2=0x7f49c40c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 --2- 172.21.15.81:0/1301301534 >> v2:172.21.15.106:3300/0 conn(0x7f49c415a040 0x7f49c40c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.013+0000 7f49c9600640 1 -- 172.21.15.81:0/1301301534 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c4072840 con 0x7f49c4072430 2024-05-27T06:59:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49c9600640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c4072430 0x7f49c4159b00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f49c406f550 tx=0x7f49b801e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79457c5c9cbb4e86 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:20.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49c3e00640 1 -- 172.21.15.81:0/1301301534 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b80b4070 con 0x7f49c4072430 2024-05-27T06:59:20.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49c3e00640 1 -- 172.21.15.81:0/1301301534 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b80a77c0 con 0x7f49c4072430 2024-05-27T06:59:20.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49c3e00640 1 -- 172.21.15.81:0/1301301534 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b80a7b90 con 0x7f49c4072430 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 -- 172.21.15.81:0/1301301534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c4072430 msgr2=0x7f49c4159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c4072430 0x7f49c4159b00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f49c406f550 tx=0x7f49b801e930 comp rx=0 tx=0).stop 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 -- 172.21.15.81:0/1301301534 shutdown_connections 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c40c60a0 0x7f49c415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 --2- 172.21.15.81:0/1301301534 >> v2:172.21.15.106:3300/0 conn(0x7f49c415a040 0x7f49c40c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 --2- 172.21.15.81:0/1301301534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c4072430 0x7f49c4159b00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 -- 172.21.15.81:0/1301301534 >> 172.21.15.81:0/1301301534 conn(0x7f49c4070700 msgr2=0x7f49c40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 -- 172.21.15.81:0/1301301534 shutdown_connections 2024-05-27T06:59:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.017+0000 7f49cc000640 1 -- 172.21.15.81:0/1301301534 wait complete. 2024-05-27T06:59:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 Processor -- start 2024-05-27T06:59:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 -- start start 2024-05-27T06:59:20.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c4072430 0x7f49c41f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49c40c60a0 0x7f49c4158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f49c40ce6f0 con 0x7f49c415a040 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f49c41595e0 con 0x7f49c40c60a0 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49cc000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f49c40c5610 con 0x7f49c4072430 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49782/0 (socket says 172.21.15.81:49782) 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 -- 172.21.15.81:0/2298909349 learned_addr learned my addr 172.21.15.81:0/2298909349 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49c9600640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c4072430 0x7f49c41f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49c8c00640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49c40c60a0 0x7f49c4158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 -- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c4072430 msgr2=0x7f49c41f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c4072430 0x7f49c41f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 -- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49c40c60a0 msgr2=0x7f49c4158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:20.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49c40c60a0 0x7f49c4158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 -- 172.21.15.81:0/2298909349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c41f67d0 con 0x7f49c415a040 2024-05-27T06:59:20.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.021+0000 7f49ca000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00eed0 tx=0x7f49bc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:20.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc018020 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49bc00bcd0 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc014640 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c414ebf0 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c41f9ae0 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49c41f69a0 con 0x7f49c415a040 2024-05-27T06:59:20.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f49bc01b020 con 0x7f49c415a040 2024-05-27T06:59:20.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 0x7f49a4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:20.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f49bc08fd10 con 0x7f49c415a040 2024-05-27T06:59:20.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.025+0000 7f49c9600640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 0x7f49a4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:20.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.029+0000 7f49c9600640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 0x7f49a4071940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f49b80b1000 tx=0x7f49b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:20.023 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.033+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f49bc022020 con 0x7f49c415a040 2024-05-27T06:59:20.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.217+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49c406a560 con 0x7f49c415a040 2024-05-27T06:59:20.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49c1600640 1 -- 172.21.15.81:0/2298909349 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f49bc05c310 con 0x7f49c415a040 2024-05-27T06:59:20.212 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:20.212 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:20.212 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:20.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 msgr2=0x7f49a4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 0x7f49a4071940 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f49b80b1000 tx=0x7f49b8008040 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 msgr2=0x7f49c4158b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00eed0 tx=0x7f49bc00b480 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 shutdown_connections 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f49a406f430 0x7f49a4071940 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49c415a040 0x7f49c4158b50 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49c40c60a0 0x7f49c4158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 --2- 172.21.15.81:0/2298909349 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49c4072430 0x7f49c41f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 >> 172.21.15.81:0/2298909349 conn(0x7f49c4070700 msgr2=0x7f49c406cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 shutdown_connections 2024-05-27T06:59:20.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:20.221+0000 7f49cc000640 1 -- 172.21.15.81:0/2298909349 wait complete. 2024-05-27T06:59:20.233 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:22.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:22.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.401+0000 7f3c3bc00640 1 Processor -- start 2024-05-27T06:59:22.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.401+0000 7f3c3bc00640 1 -- start start 2024-05-27T06:59:22.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c3bc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c340c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:22.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c3bc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c3415db00 con 0x7f3c340c6fd0 2024-05-27T06:59:22.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c3bc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c34073860 con 0x7f3c341546f0 2024-05-27T06:59:22.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c3bc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3c3415e6f0 con 0x7f3c3415ac20 2024-05-27T06:59:22.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c2be00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3c340c6fd0 0x7f3c3415a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34512/0 (socket says 172.21.15.81:34512) 2024-05-27T06:59:22.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c2be00640 1 -- 172.21.15.81:0/3460832951 learned_addr learned my addr 172.21.15.81:0/3460832951 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c39c00640 1 --2- 172.21.15.81:0/3460832951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c340c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824813688 0 0) 0x7f3c3415db00 con 0x7f3c340c6fd0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c18003530 con 0x7f3c340c6fd0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810244963 0 0) 0x7f3c34073860 con 0x7f3c341546f0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c3415db00 con 0x7f3c341546f0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3004101051 0 0) 0x7f3c18003530 con 0x7f3c340c6fd0 2024-05-27T06:59:22.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c39c00640 1 -- 172.21.15.81:0/3460832951 >> v1:172.21.15.106:6789/0 conn(0x7f3c341546f0 legacy=0x7f3c34072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c39c00640 1 -- 172.21.15.81:0/3460832951 >> v1:172.21.15.81:6789/0 conn(0x7f3c340c6fd0 legacy=0x7f3c3415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.405+0000 7f3c39c00640 1 -- 172.21.15.81:0/3460832951 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3415ddc0 con 0x7f3c3415ac20 2024-05-27T06:59:22.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c39c00640 1 --2- 172.21.15.81:0/3460832951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c340c73e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009dc0 tx=0x7f3c2c01f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=324741f1ff589cd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:22.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c0b5070 con 0x7f3c3415ac20 2024-05-27T06:59:22.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c2c0a8c30 con 0x7f3c3415ac20 2024-05-27T06:59:22.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c33e00640 1 -- 172.21.15.81:0/3460832951 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c001700 con 0x7f3c3415ac20 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 -- 172.21.15.81:0/3460832951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 msgr2=0x7f3c340c73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/3460832951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c340c73e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c009dc0 tx=0x7f3c2c01f0a0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 -- 172.21.15.81:0/3460832951 shutdown_connections 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/3460832951 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c340c73e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 -- 172.21.15.81:0/3460832951 >> 172.21.15.81:0/3460832951 conn(0x7f3c34070700 msgr2=0x7f3c340cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 -- 172.21.15.81:0/3460832951 shutdown_connections 2024-05-27T06:59:22.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.409+0000 7f3c3bc00640 1 -- 172.21.15.81:0/3460832951 wait complete. 2024-05-27T06:59:22.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 Processor -- start 2024-05-27T06:59:22.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- start start 2024-05-27T06:59:22.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:22.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c341546f0 0x7f3c341586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:22.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c341596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3415e380 con 0x7f3c340c6fd0 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3c340755b0 con 0x7f3c341546f0 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3c34154b00 con 0x7f3c3415ac20 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49786/0 (socket says 172.21.15.81:49786) 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 -- 172.21.15.81:0/4206276372 learned_addr learned my addr 172.21.15.81:0/4206276372 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39c00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c341596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c2be00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c341546f0 0x7f3c341586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 -- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 msgr2=0x7f3c341596c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c341596c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 -- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c341546f0 msgr2=0x7f3c341586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c341546f0 0x7f3c341586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 -- 172.21.15.81:0/4206276372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c340c9960 con 0x7f3c340c6fd0 2024-05-27T06:59:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c39200640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c400 tx=0x7f3c2400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24017070 con 0x7f3c340c6fd0 2024-05-27T06:59:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c24005310 con 0x7f3c340c6fd0 2024-05-27T06:59:22.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c340cc4e0 con 0x7f3c340c6fd0 2024-05-27T06:59:22.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c240054c0 con 0x7f3c340c6fd0 2024-05-27T06:59:22.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c340ccaa0 con 0x7f3c340c6fd0 2024-05-27T06:59:22.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.413+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c3406a560 con 0x7f3c340c6fd0 2024-05-27T06:59:22.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.417+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3c2401a070 con 0x7f3c340c6fd0 2024-05-27T06:59:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.417+0000 7f3c31600640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 0x7f3c10071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.417+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f3c2408e800 con 0x7f3c340c6fd0 2024-05-27T06:59:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.417+0000 7f3c2be00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 0x7f3c10071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:22.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.417+0000 7f3c2be00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 0x7f3c10071940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c001230 tx=0x7f3c1c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:22.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.421+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3c2405ae10 con 0x7f3c340c6fd0 2024-05-27T06:59:22.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c340c77c0 con 0x7f3c340c6fd0 2024-05-27T06:59:22.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c31600640 1 -- 172.21.15.81:0/4206276372 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f3c2405f110 con 0x7f3c340c6fd0 2024-05-27T06:59:22.608 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:22.608 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:22.608 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:22.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 msgr2=0x7f3c10071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 0x7f3c10071940 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c001230 tx=0x7f3c1c009040 comp rx=0 tx=0).stop 2024-05-27T06:59:22.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 msgr2=0x7f3c340c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400c400 tx=0x7f3c2400c8c0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 shutdown_connections 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3c3415ac20 0x7f3c341596c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3c1006f430 0x7f3c10071940 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3c341546f0 0x7f3c341586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 --2- 172.21.15.81:0/4206276372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c340c6fd0 0x7f3c340c9420 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 >> 172.21.15.81:0/4206276372 conn(0x7f3c34070700 msgr2=0x7f3c3415bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 shutdown_connections 2024-05-27T06:59:22.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:22.617+0000 7f3c3bc00640 1 -- 172.21.15.81:0/4206276372 wait complete. 2024-05-27T06:59:22.630 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:24.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:24.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.797+0000 7fb2cde00640 1 Processor -- start 2024-05-27T06:59:24.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.797+0000 7fb2cde00640 1 -- start start 2024-05-27T06:59:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2cde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb2c8072430 0x7fb2c8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2cde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2c80c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2cde00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c8072840 con 0x7fb2c80c60a0 2024-05-27T06:59:24.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2cde00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb2c80c5040 con 0x7fb2c8072430 2024-05-27T06:59:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2cde00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c80c75c0 con 0x7fb2c815a040 2024-05-27T06:59:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c7400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb2c80c60a0 0x7fb2c815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34526/0 (socket says 172.21.15.81:34526) 2024-05-27T06:59:24.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c7400640 1 -- 172.21.15.81:0/3488764415 learned_addr learned my addr 172.21.15.81:0/3488764415 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/3488764415 >> v2:172.21.15.106:3300/0 conn(0x7fb2c8072430 0x7fb2c8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2be000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2c80c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978159703 0 0) 0x7fb2c8072840 con 0x7fb2c80c60a0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2ac003530 con 0x7fb2c80c60a0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1166118080 0 0) 0x7fb2ac003530 con 0x7fb2c80c60a0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 msgr2=0x7fb2c80c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2c80c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 >> v2:172.21.15.106:3300/0 conn(0x7fb2c8072430 msgr2=0x7fb2c8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 --2- 172.21.15.81:0/3488764415 >> v2:172.21.15.106:3300/0 conn(0x7fb2c8072430 0x7fb2c8159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c80c5270 con 0x7fb2c80c60a0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fb2c0001700 con 0x7fb2c80c60a0 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 >> v1:172.21.15.81:6789/0 conn(0x7fb2c80c60a0 legacy=0x7fb2c815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:24.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2ac003a10 0x7fb2ac003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2ac004810 0x7fb2ac004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2ac005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80c75c0 con 0x7fb2ac003a10 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac00ba80 con 0x7fb2ac004810 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ac00bc00 con 0x7fb2c815a040 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2ac003a10 0x7fb2ac003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2be000640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2ac004810 0x7fb2ac004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c7400640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2ac005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 -- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 msgr2=0x7fb2ac005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2c815a040 0x7fb2ac005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 -- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2ac004810 msgr2=0x7fb2ac004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2ac004810 0x7fb2ac004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.801+0000 7fb2c6a00640 1 -- 172.21.15.81:0/3488764415 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac003530 con 0x7fb2ac003a10 2024-05-27T06:59:24.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2ac003a10 0x7fb2ac003e80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f90 tx=0x7fb2b801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540812ff6c0f9926 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:24.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b80b2070 con 0x7fb2ac003a10 2024-05-27T06:59:24.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b80a5a70 con 0x7fb2ac003a10 2024-05-27T06:59:24.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2c6000640 1 -- 172.21.15.81:0/3488764415 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8009c10 con 0x7fb2ac003a10 2024-05-27T06:59:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 -- 172.21.15.81:0/3488764415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2ac003a10 msgr2=0x7fb2ac003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 --2- 172.21.15.81:0/3488764415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2ac003a10 0x7fb2ac003e80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f90 tx=0x7fb2b801cc80 comp rx=0 tx=0).stop 2024-05-27T06:59:24.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2c7400640 1 -- 172.21.15.81:0/3488764415 reap_dead start 2024-05-27T06:59:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 -- 172.21.15.81:0/3488764415 shutdown_connections 2024-05-27T06:59:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 -- 172.21.15.81:0/3488764415 >> 172.21.15.81:0/3488764415 conn(0x7fb2c8070700 msgr2=0x7fb2c80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 -- 172.21.15.81:0/3488764415 shutdown_connections 2024-05-27T06:59:24.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.805+0000 7fb2cde00640 1 -- 172.21.15.81:0/3488764415 wait complete. 2024-05-27T06:59:24.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 Processor -- start 2024-05-27T06:59:24.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 -- start start 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2ac003a10 0x7fb2c8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c81e9540 0x7fb2c814f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80ce6f0 con 0x7fb2c814d0f0 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c80c7dd0 con 0x7fb2c81e9540 2024-05-27T06:59:24.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2cde00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c80c5610 con 0x7fb2ac003a10 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49808/0 (socket says 172.21.15.81:49808) 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 -- 172.21.15.81:0/746102337 learned_addr learned my addr 172.21.15.81:0/746102337 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2be000640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c81e9540 0x7fb2c814f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2ac003a10 0x7fb2c8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 -- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2ac003a10 msgr2=0x7fb2c8154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2ac003a10 0x7fb2c8154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 -- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c81e9540 msgr2=0x7fb2c814f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:24.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c81e9540 0x7fb2c814f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 -- 172.21.15.81:0/746102337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c80cd2a0 con 0x7fb2c814d0f0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.809+0000 7fb2c7400640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb2c806f550 tx=0x7fb2c000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0019070 con 0x7fb2c814d0f0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2c000c610 con 0x7fb2c814d0f0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000ca00 con 0x7fb2c814d0f0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c8157650 con 0x7fb2c814d0f0 2024-05-27T06:59:24.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c8158330 con 0x7fb2c814d0f0 2024-05-27T06:59:24.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c8157b70 con 0x7fb2c814d0f0 2024-05-27T06:59:24.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb2c001c070 con 0x7fb2c814d0f0 2024-05-27T06:59:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 0x7fb2a4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb2c008e990 con 0x7fb2c814d0f0 2024-05-27T06:59:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.813+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 0x7fb2a4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.817+0000 7fb2c6a00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 0x7fb2a4071910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f90 tx=0x7fb2b80a7040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:24.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:24.821+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb2c005b680 con 0x7fb2c814d0f0 2024-05-27T06:59:25.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.009+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2c806a560 con 0x7fb2c814d0f0 2024-05-27T06:59:25.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2bfe00640 1 -- 172.21.15.81:0/746102337 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fb2c005af80 con 0x7fb2c814d0f0 2024-05-27T06:59:25.004 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:25.004 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:25.004 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 msgr2=0x7fb2a4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 0x7fb2a4071910 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009f90 tx=0x7fb2b80a7040 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 msgr2=0x7fb2c814d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb2c806f550 tx=0x7fb2c000eb90 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 shutdown_connections 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb2a406f400 0x7fb2a4071910 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c814d0f0 0x7fb2c814d500 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb2c81e9540 0x7fb2c814f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 --2- 172.21.15.81:0/746102337 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb2ac003a10 0x7fb2c8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 >> 172.21.15.81:0/746102337 conn(0x7fb2c8070700 msgr2=0x7fb2c81505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:25.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 shutdown_connections 2024-05-27T06:59:25.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:25.013+0000 7fb2cde00640 1 -- 172.21.15.81:0/746102337 wait complete. 2024-05-27T06:59:25.025 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:25.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~201s 2024-05-27T06:59:27.027 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:27.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.192+0000 7f3b45e00640 1 Processor -- start 2024-05-27T06:59:27.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.192+0000 7f3b45e00640 1 -- start start 2024-05-27T06:59:27.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b45e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b40072430 0x7f3b40159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b401640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b45e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b40072840 con 0x7f3b4015a040 2024-05-27T06:59:27.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b45e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3b400c5040 con 0x7f3b40072430 2024-05-27T06:59:27.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b45e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3b400c75c0 con 0x7f3b40161cd0 2024-05-27T06:59:27.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b40072430 0x7f3b40159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b36000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3b4015a040 0x7f3b400c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51618/0 (socket says 172.21.15.81:51618) 2024-05-27T06:59:27.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b36000640 1 -- 172.21.15.81:0/1610111686 learned_addr learned my addr 172.21.15.81:0/1610111686 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3f400640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b401640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3ea00640 1 -- 172.21.15.81:0/1610111686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 msgr2=0x7f3b401640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b401640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3ea00640 1 -- 172.21.15.81:0/1610111686 >> v1:172.21.15.81:6789/0 conn(0x7f3b4015a040 legacy=0x7f3b400c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3ea00640 1 -- 172.21.15.81:0/1610111686 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b400c5270 con 0x7f3b40072430 2024-05-27T06:59:27.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.196+0000 7f3b3e000640 1 -- 172.21.15.81:0/1610111686 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699769717 0 0) 0x7f3b40072840 con 0x7f3b4015a040 2024-05-27T06:59:27.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b40072430 0x7f3b40159b00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3b300097c0 tx=0x7f3b3001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1109c9bc6f10191e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:27.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b3e000640 1 -- 172.21.15.81:0/1610111686 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b300b0070 con 0x7f3b40072430 2024-05-27T06:59:27.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b3e000640 1 -- 172.21.15.81:0/1610111686 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b300ab600 con 0x7f3b40072430 2024-05-27T06:59:27.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b3e000640 1 -- 172.21.15.81:0/1610111686 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b300ab9d0 con 0x7f3b40072430 2024-05-27T06:59:27.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- 172.21.15.81:0/1610111686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b40072430 msgr2=0x7f3b40159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b40072430 0x7f3b40159b00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3b300097c0 tx=0x7f3b3001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- 172.21.15.81:0/1610111686 shutdown_connections 2024-05-27T06:59:27.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b401640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.192 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 --2- 172.21.15.81:0/1610111686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b40072430 0x7f3b40159b00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- 172.21.15.81:0/1610111686 >> 172.21.15.81:0/1610111686 conn(0x7f3b40070700 msgr2=0x7f3b400cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:27.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- 172.21.15.81:0/1610111686 shutdown_connections 2024-05-27T06:59:27.193 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- 172.21.15.81:0/1610111686 wait complete. 2024-05-27T06:59:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 Processor -- start 2024-05-27T06:59:27.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.200+0000 7f3b45e00640 1 -- start start 2024-05-27T06:59:27.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 0x7f3b40149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b40147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b400ce6f0 con 0x7f3b4015a040 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b400c7dd0 con 0x7f3b40072430 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3b400c5610 con 0x7f3b40161cd0 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45340/0 (socket says 172.21.15.81:45340) 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 -- 172.21.15.81:0/4205545288 learned_addr learned my addr 172.21.15.81:0/4205545288 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b3f400640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b40147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 0x7f3b40149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 -- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 msgr2=0x7f3b40147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b40147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 -- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 msgr2=0x7f3b40149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 0x7f3b40149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 -- 172.21.15.81:0/4205545288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b40158210 con 0x7f3b4015a040 2024-05-27T06:59:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b3f400640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b40147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:59:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 0x7f3b40149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:59:27.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b36000640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c400 tx=0x7f3b2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b280074e0 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b280047f0 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b40157450 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28007680 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b40157a10 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.204+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b4006a560 con 0x7f3b4015a040 2024-05-27T06:59:27.199 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.208+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3b28011070 con 0x7f3b4015a040 2024-05-27T06:59:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.208+0000 7f3b37e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 0x7f3b1c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.208+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f3b2808f060 con 0x7f3b4015a040 2024-05-27T06:59:27.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.208+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 0x7f3b1c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:27.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.208+0000 7f3b3ea00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 0x7f3b1c071940 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b300ad000 tx=0x7f3b30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:27.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.212+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3b2805bcc0 con 0x7f3b4015a040 2024-05-27T06:59:27.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.400+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b400c8f60 con 0x7f3b4015a040 2024-05-27T06:59:27.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.400+0000 7f3b37e00640 1 -- 172.21.15.81:0/4205545288 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f3b2805b5c0 con 0x7f3b4015a040 2024-05-27T06:59:27.395 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:27.395 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:27.395 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:27.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 msgr2=0x7f3b1c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 0x7f3b1c071940 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3b300ad000 tx=0x7f3b30008040 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 msgr2=0x7f3b4014cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c400 tx=0x7f3b2800c8c0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 shutdown_connections 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3b40161cd0 0x7f3b40147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3b1c06f430 0x7f3b1c071940 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b4015a040 0x7f3b4014cdd0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 --2- 172.21.15.81:0/4205545288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b40072430 0x7f3b40149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 >> 172.21.15.81:0/4205545288 conn(0x7f3b40070700 msgr2=0x7f3b400cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 shutdown_connections 2024-05-27T06:59:27.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:27.404+0000 7f3b45e00640 1 -- 172.21.15.81:0/4205545288 wait complete. 2024-05-27T06:59:27.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:29.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:29.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.584+0000 7fbc56800640 1 Processor -- start 2024-05-27T06:59:29.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.584+0000 7fbc56800640 1 -- start start 2024-05-27T06:59:29.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc56800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc50072430 0x7fbc50159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc56800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fbc5015a040 0x7fbc500c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc56800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc50072840 con 0x7fbc500c60a0 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc56800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fbc500c5040 con 0x7fbc50072430 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc56800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fbc500c75c0 con 0x7fbc5015a040 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc50072430 0x7fbc50159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc50072430 0x7fbc50159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45350/0 (socket says 172.21.15.81:45350) 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 -- 172.21.15.81:0/3569798588 learned_addr learned my addr 172.21.15.81:0/3569798588 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 -- 172.21.15.81:0/3569798588 >> v2:172.21.15.106:3300/0 conn(0x7fbc5015a040 msgr2=0x7fbc500c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 --2- 172.21.15.81:0/3569798588 >> v2:172.21.15.106:3300/0 conn(0x7fbc5015a040 0x7fbc500c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 -- 172.21.15.81:0/3569798588 >> v1:172.21.15.106:6789/0 conn(0x7fbc500c60a0 legacy=0x7fbc5015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:29.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.588+0000 7fbc4f400640 1 -- 172.21.15.81:0/3569798588 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc500c5270 con 0x7fbc50072430 2024-05-27T06:59:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc4f400640 1 --2- 172.21.15.81:0/3569798588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50159b00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbc5006f550 tx=0x7fbc4001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e680147e2eee4c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc4ea00640 1 -- 172.21.15.81:0/3569798588 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400a5590 con 0x7fbc50072430 2024-05-27T06:59:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc4ea00640 1 -- 172.21.15.81:0/3569798588 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc400a5730 con 0x7fbc50072430 2024-05-27T06:59:29.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc4ea00640 1 -- 172.21.15.81:0/3569798588 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400a5b00 con 0x7fbc50072430 2024-05-27T06:59:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 -- 172.21.15.81:0/3569798588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 msgr2=0x7fbc50159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 --2- 172.21.15.81:0/3569798588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50159b00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbc5006f550 tx=0x7fbc4001c9f0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 -- 172.21.15.81:0/3569798588 shutdown_connections 2024-05-27T06:59:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 --2- 172.21.15.81:0/3569798588 >> v2:172.21.15.106:3300/0 conn(0x7fbc5015a040 0x7fbc500c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 --2- 172.21.15.81:0/3569798588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50159b00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 -- 172.21.15.81:0/3569798588 >> 172.21.15.81:0/3569798588 conn(0x7fbc50070700 msgr2=0x7fbc500cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 -- 172.21.15.81:0/3569798588 shutdown_connections 2024-05-27T06:59:29.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.592+0000 7fbc56800640 1 -- 172.21.15.81:0/3569798588 wait complete. 2024-05-27T06:59:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 Processor -- start 2024-05-27T06:59:29.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- start start 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 0x7fbc5013f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc5015a040 0x7fbc5013fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc500ce6f0 con 0x7fbc50072430 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbc500c7dd0 con 0x7fbc500c60a0 2024-05-27T06:59:29.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fbc500c5610 con 0x7fbc5015a040 2024-05-27T06:59:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45358/0 (socket says 172.21.15.81:45358) 2024-05-27T06:59:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 -- 172.21.15.81:0/2126650296 learned_addr learned my addr 172.21.15.81:0/2126650296 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:29.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc46a00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 0x7fbc5013f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4fe00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc5015a040 0x7fbc5013fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 -- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc5015a040 msgr2=0x7fbc5013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc5015a040 0x7fbc5013fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 -- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 msgr2=0x7fbc5013f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 0x7fbc5013f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 -- 172.21.15.81:0/2126650296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc50154bd0 con 0x7fbc50072430 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc46a00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 0x7fbc5013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:29.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc4f400640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbc400af040 tx=0x7fbc40019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400b8670 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc400be070 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5013e4d0 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400b8a90 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.596+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5013ea70 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5006a560 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fbc400b8c30 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc47e00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 0x7fbc2c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fbc4012b1a0 con 0x7fbc50072430 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc46a00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 0x7fbc2c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.600+0000 7fbc46a00640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 0x7fbc2c071910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbc38006fd0 tx=0x7fbc38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:29.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.604+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fbc400165e0 con 0x7fbc50072430 2024-05-27T06:59:29.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.796+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc500c5940 con 0x7fbc50072430 2024-05-27T06:59:29.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc47e00640 1 -- 172.21.15.81:0/2126650296 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fbc400fee20 con 0x7fbc50072430 2024-05-27T06:59:29.791 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:29.791 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:29.792 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:29.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 msgr2=0x7fbc2c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 0x7fbc2c071910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbc38006fd0 tx=0x7fbc38008040 comp rx=0 tx=0).stop 2024-05-27T06:59:29.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 msgr2=0x7fbc50154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fbc400af040 tx=0x7fbc40019790 comp rx=0 tx=0).stop 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 shutdown_connections 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fbc5015a040 0x7fbc5013fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fbc2c06f400 0x7fbc2c071910 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc500c60a0 0x7fbc5013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 --2- 172.21.15.81:0/2126650296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50072430 0x7fbc50154690 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 >> 172.21.15.81:0/2126650296 conn(0x7fbc50070700 msgr2=0x7fbc500c4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:29.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 shutdown_connections 2024-05-27T06:59:29.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:29.800+0000 7fbc56800640 1 -- 172.21.15.81:0/2126650296 wait complete. 2024-05-27T06:59:29.814 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:59:31.815 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:31.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.980+0000 7f104fe00640 1 Processor -- start 2024-05-27T06:59:31.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.980+0000 7f104fe00640 1 -- start start 2024-05-27T06:59:31.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104fe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104fe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104fe00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1048072840 con 0x7f104815a040 2024-05-27T06:59:31.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104fe00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f10480c5040 con 0x7f1048072430 2024-05-27T06:59:31.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104fe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f10480c75c0 con 0x7f1048161cd0 2024-05-27T06:59:31.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:54892/0 (socket says 172.21.15.81:54892) 2024-05-27T06:59:31.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:39276/0 (socket says 172.21.15.81:39276) 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104d400640 1 -- 172.21.15.81:0/1668393908 learned_addr learned my addr 172.21.15.81:0/1668393908 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104ca00640 1 --1- 172.21.15.81:0/1668393908 >> v1:172.21.15.106:6789/0 conn(0x7f104815a040 0x7f10480c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:44412/0 (socket says 172.21.15.81:44412) 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 -- 172.21.15.81:0/1668393908 >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 msgr2=0x7f1048159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f1047e00640 1 -- 172.21.15.81:0/1668393908 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3562739552 0 0) 0x7f1048072840 con 0x7f104815a040 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 --2- 172.21.15.81:0/1668393908 >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 -- 172.21.15.81:0/1668393908 >> v1:172.21.15.106:6789/0 conn(0x7f104815a040 legacy=0x7f10480c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104de00640 1 -- 172.21.15.81:0/1668393908 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10480c5270 con 0x7f1048161cd0 2024-05-27T06:59:31.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.984+0000 7f104d400640 1 --2- 172.21.15.81:0/1668393908 >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T06:59:31.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104de00640 1 --2- 172.21.15.81:0/1668393908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1040009d90 tx=0x7f104001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28087c646a1c0eea server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:31.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f1047e00640 1 -- 172.21.15.81:0/1668393908 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400b2070 con 0x7f1048161cd0 2024-05-27T06:59:31.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f1047e00640 1 -- 172.21.15.81:0/1668393908 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10400a5730 con 0x7f1048161cd0 2024-05-27T06:59:31.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f1047e00640 1 -- 172.21.15.81:0/1668393908 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400a5b00 con 0x7f1048161cd0 2024-05-27T06:59:31.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 -- 172.21.15.81:0/1668393908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 msgr2=0x7f10481640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:31.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 --2- 172.21.15.81:0/1668393908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1040009d90 tx=0x7f104001c970 comp rx=0 tx=0).stop 2024-05-27T06:59:31.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 -- 172.21.15.81:0/1668393908 shutdown_connections 2024-05-27T06:59:31.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 --2- 172.21.15.81:0/1668393908 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1048161cd0 0x7f10481640f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:31.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 --2- 172.21.15.81:0/1668393908 >> v2:172.21.15.106:3300/0 conn(0x7f1048072430 0x7f1048159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:31.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 -- 172.21.15.81:0/1668393908 >> 172.21.15.81:0/1668393908 conn(0x7f1048070700 msgr2=0x7f10480cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:31.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 -- 172.21.15.81:0/1668393908 shutdown_connections 2024-05-27T06:59:31.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.988+0000 7f104fe00640 1 -- 172.21.15.81:0/1668393908 wait complete. 2024-05-27T06:59:31.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 Processor -- start 2024-05-27T06:59:31.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- start start 2024-05-27T06:59:31.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1048072430 0x7f104814c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f104815a040 0x7f1048158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f10480ce6f0 con 0x7f1048161cd0 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f10480c7dd0 con 0x7f1048072430 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f10480c5610 con 0x7f104815a040 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45370/0 (socket says 172.21.15.81:45370) 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 -- 172.21.15.81:0/2294566508 learned_addr learned my addr 172.21.15.81:0/2294566508 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104ca00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f104815a040 0x7f1048158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104d400640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1048072430 0x7f104814c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 -- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f104815a040 msgr2=0x7f1048158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:31.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f104815a040 0x7f1048158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:31.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 -- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1048072430 msgr2=0x7f104814c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1048072430 0x7f104814c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 -- 172.21.15.81:0/2294566508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104814cd50 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104de00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f10400b0040 tx=0x7f1040019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400b2050 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10400be070 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481f6730 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400ad890 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.992+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10481543c0 con 0x7f1048161cd0 2024-05-27T06:59:31.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.996+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104806a560 con 0x7f1048161cd0 2024-05-27T06:59:31.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.996+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f10400ada30 con 0x7f1048161cd0 2024-05-27T06:59:31.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.996+0000 7f1045600640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 0x7f1028071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:31.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.996+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f104012ad80 con 0x7f1048161cd0 2024-05-27T06:59:31.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:31.996+0000 7f104d400640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 0x7f1028071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:31.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.000+0000 7f104d400640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 0x7f1028071910 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f103c00b240 tx=0x7f103c00b450 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:31.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.000+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1040016560 con 0x7f1048161cd0 2024-05-27T06:59:32.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10480c5940 con 0x7f1048161cd0 2024-05-27T06:59:32.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f1045600640 1 -- 172.21.15.81:0/2294566508 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f10400f7510 con 0x7f1048161cd0 2024-05-27T06:59:32.184 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:32.184 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:32.184 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:32.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 msgr2=0x7f1028071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:32.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 0x7f1028071910 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f103c00b240 tx=0x7f103c00b450 comp rx=0 tx=0).stop 2024-05-27T06:59:32.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 msgr2=0x7f1048148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:32.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f10400b0040 tx=0x7f1040019710 comp rx=0 tx=0).stop 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 shutdown_connections 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f104815a040 0x7f1048158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f102806f400 0x7f1028071910 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1048161cd0 0x7f1048148130 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 --2- 172.21.15.81:0/2294566508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1048072430 0x7f104814c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 >> 172.21.15.81:0/2294566508 conn(0x7f1048070700 msgr2=0x7f10480c39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 shutdown_connections 2024-05-27T06:59:32.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:32.192+0000 7f104fe00640 1 -- 172.21.15.81:0/2294566508 wait complete. 2024-05-27T06:59:32.206 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:32.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~208s 2024-05-27T06:59:34.207 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.372+0000 7f201c000640 1 Processor -- start 2024-05-27T06:59:34.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 -- start start 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141606e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20140caca0 0x7f20140cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2014160c20 con 0x7f20140d02a0 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f201406eb20 con 0x7f20140caca0 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201c000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f201406f910 con 0x7f20140ca2f0 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2018c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20140caca0 0x7f20140cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201a000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f20140d02a0 0x7f20140cc2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51632/0 (socket says 172.21.15.81:51632) 2024-05-27T06:59:34.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f201a000640 1 -- 172.21.15.81:0/1182452584 learned_addr learned my addr 172.21.15.81:0/1182452584 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:34.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2019600640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141606e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2013e00640 1 -- 172.21.15.81:0/1182452584 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980386325 0 0) 0x7f2014160c20 con 0x7f20140d02a0 2024-05-27T06:59:34.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2013e00640 1 -- 172.21.15.81:0/1182452584 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ffc003530 con 0x7f20140d02a0 2024-05-27T06:59:34.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2018c00640 1 -- 172.21.15.81:0/1182452584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 msgr2=0x7f20141606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:34.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2018c00640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141606e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2018c00640 1 -- 172.21.15.81:0/1182452584 >> v1:172.21.15.81:6789/0 conn(0x7f20140d02a0 legacy=0x7f20140cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.376+0000 7f2018c00640 1 -- 172.21.15.81:0/1182452584 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201415c710 con 0x7f20140caca0 2024-05-27T06:59:34.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f2018c00640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140caca0 0x7f20140cfd60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f20000097c0 tx=0x7f200001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef08343a4b43ca5 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:34.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f2013e00640 1 -- 172.21.15.81:0/1182452584 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000b0070 con 0x7f20140caca0 2024-05-27T06:59:34.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f2013e00640 1 -- 172.21.15.81:0/1182452584 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20000ab600 con 0x7f20140caca0 2024-05-27T06:59:34.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f2013e00640 1 -- 172.21.15.81:0/1182452584 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000ab9d0 con 0x7f20140caca0 2024-05-27T06:59:34.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 -- 172.21.15.81:0/1182452584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140caca0 msgr2=0x7f20140cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140caca0 0x7f20140cfd60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f20000097c0 tx=0x7f200001c4a0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 -- 172.21.15.81:0/1182452584 shutdown_connections 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140caca0 0x7f20140cfd60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 --2- 172.21.15.81:0/1182452584 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141606e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 -- 172.21.15.81:0/1182452584 >> 172.21.15.81:0/1182452584 conn(0x7f2014068ae0 msgr2=0x7f2014068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 -- 172.21.15.81:0/1182452584 shutdown_connections 2024-05-27T06:59:34.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.380+0000 7f201c000640 1 -- 172.21.15.81:0/1182452584 wait complete. 2024-05-27T06:59:34.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 Processor -- start 2024-05-27T06:59:34.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- start start 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 0x7f2014155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f201415cf90 con 0x7f20140d02a0 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f201415ccb0 con 0x7f20140caca0 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f201406e220 con 0x7f20140ca2f0 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45392/0 (socket says 172.21.15.81:45392) 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 -- 172.21.15.81:0/863312161 learned_addr learned my addr 172.21.15.81:0/863312161 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:34.379 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2018c00640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 0x7f2014155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2019600640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 -- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 msgr2=0x7f20141584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 -- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 msgr2=0x7f2014155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 0x7f2014155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 -- 172.21.15.81:0/863312161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2014158a20 con 0x7f20140d02a0 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2018c00640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 0x7f2014155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201a000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f200c00cad0 tx=0x7f200c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:34.380 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c00ac30 con 0x7f20140d02a0 2024-05-27T06:59:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200c00add0 con 0x7f20140d02a0 2024-05-27T06:59:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c012600 con 0x7f20140d02a0 2024-05-27T06:59:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20141f6ca0 con 0x7f20140d02a0 2024-05-27T06:59:34.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.384+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2014159d30 con 0x7f20140d02a0 2024-05-27T06:59:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201406a560 con 0x7f20140d02a0 2024-05-27T06:59:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f200c0127a0 con 0x7f20140d02a0 2024-05-27T06:59:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f2011600640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 0x7f1ff4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f200c08fb30 con 0x7f20140d02a0 2024-05-27T06:59:34.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f2019600640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 0x7f1ff4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:34.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.388+0000 7f2019600640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 0x7f1ff4071940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2008009a50 tx=0x7f2008008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:34.386 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.392+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f200c05c790 con 0x7f20140d02a0 2024-05-27T06:59:34.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20140c9680 con 0x7f20140d02a0 2024-05-27T06:59:34.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f2011600640 1 -- 172.21.15.81:0/863312161 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f200c05c090 con 0x7f20140d02a0 2024-05-27T06:59:34.576 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:34.576 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:34.576 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 msgr2=0x7f1ff4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 0x7f1ff4071940 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2008009a50 tx=0x7f2008008040 comp rx=0 tx=0).stop 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 msgr2=0x7f2014155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f200c00cad0 tx=0x7f200c00cf90 comp rx=0 tx=0).stop 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 shutdown_connections 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1ff406f430 0x7f1ff4071940 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20140d02a0 0x7f2014155640 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f20140caca0 0x7f2014155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 --2- 172.21.15.81:0/863312161 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f20140ca2f0 0x7f20141584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:34.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 >> 172.21.15.81:0/863312161 conn(0x7f2014068ae0 msgr2=0x7f20140ce1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:34.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 shutdown_connections 2024-05-27T06:59:34.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:34.584+0000 7f201c000640 1 -- 172.21.15.81:0/863312161 wait complete. 2024-05-27T06:59:34.597 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:35.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:59:35.058 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T06:59:36.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:36.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.764+0000 7f196e400640 1 Processor -- start 2024-05-27T06:59:36.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.764+0000 7f196e400640 1 -- start start 2024-05-27T06:59:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f196e400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1968072430 0x7f1968159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f196e400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f196815a040 0x7f19680c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f196e400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1968072840 con 0x7f19680c60a0 2024-05-27T06:59:36.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f196e400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f19680c5040 con 0x7f196815a040 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f196e400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f19680c75c0 con 0x7f1968072430 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f196815a040 0x7f19680c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f196815a040 0x7f19680c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34164/0 (socket says 172.21.15.81:34164) 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 -- 172.21.15.81:0/3177771008 learned_addr learned my addr 172.21.15.81:0/3177771008 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1967400640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1968072430 0x7f1968159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 -- 172.21.15.81:0/3177771008 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1968072430 msgr2=0x7f1968159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1968072430 0x7f1968159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 -- 172.21.15.81:0/3177771008 >> v1:172.21.15.106:6789/0 conn(0x7f19680c60a0 legacy=0x7f196815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.768+0000 7f1966a00640 1 -- 172.21.15.81:0/3177771008 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19680c5270 con 0x7f196815a040 2024-05-27T06:59:36.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f1966a00640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f196815a040 0x7f19680c5b60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1954009870 tx=0x7f195401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9429bc7c2fb7d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:36.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f1966000640 1 -- 172.21.15.81:0/3177771008 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540b0070 con 0x7f196815a040 2024-05-27T06:59:36.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f1966000640 1 -- 172.21.15.81:0/3177771008 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19540ab600 con 0x7f196815a040 2024-05-27T06:59:36.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f1966000640 1 -- 172.21.15.81:0/3177771008 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19540ab9d0 con 0x7f196815a040 2024-05-27T06:59:36.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 -- 172.21.15.81:0/3177771008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f196815a040 msgr2=0x7f19680c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f196815a040 0x7f19680c5b60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1954009870 tx=0x7f195401c560 comp rx=0 tx=0).stop 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 -- 172.21.15.81:0/3177771008 shutdown_connections 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f196815a040 0x7f19680c5b60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 --2- 172.21.15.81:0/3177771008 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1968072430 0x7f1968159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 -- 172.21.15.81:0/3177771008 >> 172.21.15.81:0/3177771008 conn(0x7f1968070700 msgr2=0x7f19680cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 -- 172.21.15.81:0/3177771008 shutdown_connections 2024-05-27T06:59:36.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.772+0000 7f196e400640 1 -- 172.21.15.81:0/3177771008 wait complete. 2024-05-27T06:59:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 Processor -- start 2024-05-27T06:59:36.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- start start 2024-05-27T06:59:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 0x7f196814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 0x7f19681526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19680ce6f0 con 0x7f1968072430 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f19680c7dd0 con 0x7f196815a040 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f19680c5610 con 0x7f19680c60a0 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34180/0 (socket says 172.21.15.81:34180) 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 -- 172.21.15.81:0/2732007991 learned_addr learned my addr 172.21.15.81:0/2732007991 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1966a00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 0x7f196814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967e00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 0x7f19681526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 -- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 msgr2=0x7f196814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 0x7f196814d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 -- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 msgr2=0x7f19681526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 0x7f19681526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 -- 172.21.15.81:0/2732007991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19681f7bc0 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967e00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 0x7f19681526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1966a00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 0x7f196814d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f1967400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f195800d5f0 tx=0x7f195800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580125f0 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1958019070 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580129d0 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681554d0 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.776+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1968151920 con 0x7f1968072430 2024-05-27T06:59:36.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196806a560 con 0x7f1968072430 2024-05-27T06:59:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1958012b70 con 0x7f1968072430 2024-05-27T06:59:36.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f195f400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 0x7f1948071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f195808f7b0 con 0x7f1968072430 2024-05-27T06:59:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f1966a00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 0x7f1948071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:36.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.780+0000 7f1966a00640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 0x7f1948071940 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f19540ad000 tx=0x7f1954008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:36.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.784+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1958023020 con 0x7f1968072430 2024-05-27T06:59:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.976+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19681f72a0 con 0x7f1968072430 2024-05-27T06:59:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.976+0000 7f195f400640 1 -- 172.21.15.81:0/2732007991 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f195805c410 con 0x7f1968072430 2024-05-27T06:59:36.971 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:36.971 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:36.971 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:36.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 msgr2=0x7f1948071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 0x7f1948071940 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f19540ad000 tx=0x7f1954008040 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 msgr2=0x7f19681f7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f195800d5f0 tx=0x7f195800dab0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 shutdown_connections 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19680c60a0 0x7f196814d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f194806f430 0x7f1948071940 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f196815a040 0x7f19681526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 --2- 172.21.15.81:0/2732007991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1968072430 0x7f19681f7680 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 >> 172.21.15.81:0/2732007991 conn(0x7f1968070700 msgr2=0x7f19680c4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 shutdown_connections 2024-05-27T06:59:36.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:36.980+0000 7f196e400640 1 -- 172.21.15.81:0/2732007991 wait complete. 2024-05-27T06:59:36.993 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:38.993 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 Processor -- start 2024-05-27T06:59:39.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 -- start start 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f82240ca1e0 0x7f8224156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82240c9df0 con 0x7f8224075200 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f822415d7e0 con 0x7f8224071f90 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.156+0000 7f8229800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8224074660 con 0x7f82240ca1e0 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8228e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8224075200 0x7f8224167170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50184/0 (socket says 172.21.15.81:50184) 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8228e00640 1 -- 172.21.15.81:0/1352336263 learned_addr learned my addr 172.21.15.81:0/1352336263 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:39.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223e00640 1 --2- 172.21.15.81:0/1352336263 >> v2:172.21.15.81:3300/0 conn(0x7f82240ca1e0 0x7f8224156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:39.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223400640 1 -- 172.21.15.81:0/1352336263 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454196785 0 0) 0x7f821c0014a0 con 0x7f8224075200 2024-05-27T06:59:39.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223400640 1 -- 172.21.15.81:0/1352336263 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8208003530 con 0x7f8224075200 2024-05-27T06:59:39.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223e00640 1 -- 172.21.15.81:0/1352336263 >> v1:172.21.15.106:6789/0 conn(0x7f8224071f90 legacy=0x7f8224074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223e00640 1 -- 172.21.15.81:0/1352336263 >> v1:172.21.15.81:6789/0 conn(0x7f8224075200 legacy=0x7f8224167170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223e00640 1 -- 172.21.15.81:0/1352336263 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822415cc60 con 0x7f82240ca1e0 2024-05-27T06:59:39.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223e00640 1 --2- 172.21.15.81:0/1352336263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f8224156cc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f82140097a0 tx=0x7f821401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2285fb130a35f334 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:39.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223400640 1 -- 172.21.15.81:0/1352336263 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82140b1070 con 0x7f82240ca1e0 2024-05-27T06:59:39.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223400640 1 -- 172.21.15.81:0/1352336263 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82140ac620 con 0x7f82240ca1e0 2024-05-27T06:59:39.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.160+0000 7f8223400640 1 -- 172.21.15.81:0/1352336263 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82140ac9f0 con 0x7f82240ca1e0 2024-05-27T06:59:39.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- 172.21.15.81:0/1352336263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 msgr2=0x7f8224156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 --2- 172.21.15.81:0/1352336263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f8224156cc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f82140097a0 tx=0x7f821401c3c0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- 172.21.15.81:0/1352336263 shutdown_connections 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 --2- 172.21.15.81:0/1352336263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f8224156cc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- 172.21.15.81:0/1352336263 >> 172.21.15.81:0/1352336263 conn(0x7f8224068cb0 msgr2=0x7f82240cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- 172.21.15.81:0/1352336263 shutdown_connections 2024-05-27T06:59:39.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- 172.21.15.81:0/1352336263 wait complete. 2024-05-27T06:59:39.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 Processor -- start 2024-05-27T06:59:39.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.164+0000 7f8229800640 1 -- start start 2024-05-27T06:59:39.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 0x7f82240c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 0x7f8224157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f822415c890 con 0x7f82240ca1e0 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82240727b0 con 0x7f8224075200 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f822415ca10 con 0x7f8224071f90 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34200/0 (socket says 172.21.15.81:34200) 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 -- 172.21.15.81:0/4170661458 learned_addr learned my addr 172.21.15.81:0/4170661458 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8223e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 0x7f82240c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f821b400640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 0x7f8224157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 -- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 msgr2=0x7f82240c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 0x7f82240c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 -- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 msgr2=0x7f8224157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 0x7f8224157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 -- 172.21.15.81:0/4170661458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82240c4890 con 0x7f82240ca1e0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8223e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 0x7f82240c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f821b400640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 0x7f8224157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8228e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f821c00caa0 tx=0x7f821c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c00ac00 con 0x7f82240ca1e0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c00ada0 con 0x7f82240ca1e0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8224156cc0 con 0x7f82240ca1e0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c012620 con 0x7f82240ca1e0 2024-05-27T06:59:39.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82240c6760 con 0x7f82240ca1e0 2024-05-27T06:59:39.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~215s 2024-05-27T06:59:39.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8224050870 con 0x7f82240ca1e0 2024-05-27T06:59:39.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.168+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f821c0127c0 con 0x7f82240ca1e0 2024-05-27T06:59:39.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.172+0000 7f8220c00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 0x7f8200071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:39.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.172+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f821c08fb80 con 0x7f82240ca1e0 2024-05-27T06:59:39.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.172+0000 7f8223e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 0x7f8200071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:39.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.172+0000 7f8223e00640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 0x7f8200071940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f82140ae000 tx=0x7f82140a7000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:39.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.176+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f82240c9df0 con 0x7f82240ca1e0 2024-05-27T06:59:39.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.364+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82240c3f70 con 0x7f82240ca1e0 2024-05-27T06:59:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.364+0000 7f8220c00640 1 -- 172.21.15.81:0/4170661458 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f821c08f020 con 0x7f82240ca1e0 2024-05-27T06:59:39.358 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:39.358 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:39.359 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 msgr2=0x7f8200071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 0x7f8200071940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f82140ae000 tx=0x7f82140a7000 comp rx=0 tx=0).stop 2024-05-27T06:59:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 msgr2=0x7f82241522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f821c00caa0 tx=0x7f821c00cf60 comp rx=0 tx=0).stop 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 shutdown_connections 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f820006f430 0x7f8200071940 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82240ca1e0 0x7f82241522a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8224075200 0x7f8224157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 --2- 172.21.15.81:0/4170661458 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8224071f90 0x7f82240c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 >> 172.21.15.81:0/4170661458 conn(0x7f8224068cb0 msgr2=0x7f82240cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 shutdown_connections 2024-05-27T06:59:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:39.368+0000 7f8229800640 1 -- 172.21.15.81:0/4170661458 wait complete. 2024-05-27T06:59:39.380 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:41.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:41.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 Processor -- start 2024-05-27T06:59:41.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 -- start start 2024-05-27T06:59:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 0x7f08e4072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08e40c9c70 con 0x7f08e40c9630 2024-05-27T06:59:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f08e40c9a40 con 0x7f08e4154780 2024-05-27T06:59:41.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e9800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f08e415b0c0 con 0x7f08e4072a60 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e3400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 0x7f08e4072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:59704/0 (socket says 172.21.15.81:59704) 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e3400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 0x7f08e4072520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:46032/0 (socket says 172.21.15.81:46032) 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 -- 172.21.15.81:0/2776056394 learned_addr learned my addr 172.21.15.81:0/2776056394 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e3e00640 1 --1- 172.21.15.81:0/2776056394 >> v1:172.21.15.106:6789/0 conn(0x7f08e40c9630 0x7f08e415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:50190/0 (socket says 172.21.15.81:50190) 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 -- 172.21.15.81:0/2776056394 >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 msgr2=0x7f08e4072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:41.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 --2- 172.21.15.81:0/2776056394 >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 0x7f08e4072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 -- 172.21.15.81:0/2776056394 >> v1:172.21.15.106:6789/0 conn(0x7f08e40c9630 legacy=0x7f08e415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.548+0000 7f08e8e00640 1 -- 172.21.15.81:0/2776056394 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e415a4d0 con 0x7f08e4072a60 2024-05-27T06:59:41.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e8e00640 1 --2- 172.21.15.81:0/2776056394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009e50 tx=0x7f08dc01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2adbaf15ce194bf9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e2a00640 1 -- 172.21.15.81:0/2776056394 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0a5810 con 0x7f08e4072a60 2024-05-27T06:59:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e2a00640 1 -- 172.21.15.81:0/2776056394 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc0a59b0 con 0x7f08e4072a60 2024-05-27T06:59:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e2a00640 1 -- 172.21.15.81:0/2776056394 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0a5d80 con 0x7f08e4072a60 2024-05-27T06:59:41.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 -- 172.21.15.81:0/2776056394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 msgr2=0x7f08e4164e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 --2- 172.21.15.81:0/2776056394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009e50 tx=0x7f08dc01ca30 comp rx=0 tx=0).stop 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 -- 172.21.15.81:0/2776056394 shutdown_connections 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 --2- 172.21.15.81:0/2776056394 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e4072a60 0x7f08e4164e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 --2- 172.21.15.81:0/2776056394 >> v2:172.21.15.106:3300/0 conn(0x7f08e4154780 0x7f08e4072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 -- 172.21.15.81:0/2776056394 >> 172.21.15.81:0/2776056394 conn(0x7f08e4068880 msgr2=0x7f08e4068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 -- 172.21.15.81:0/2776056394 shutdown_connections 2024-05-27T06:59:41.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.552+0000 7f08e9800640 1 -- 172.21.15.81:0/2776056394 wait complete. 2024-05-27T06:59:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 Processor -- start 2024-05-27T06:59:41.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 -- start start 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 0x7f08e414e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e4154780 0x7f08e4151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f08e40cd280 con 0x7f08e4072a60 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08e415ad50 con 0x7f08e4154780 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e9800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f08e415e1d0 con 0x7f08e40c9630 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34216/0 (socket says 172.21.15.81:34216) 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.556+0000 7f08e3e00640 1 -- 172.21.15.81:0/3143698920 learned_addr learned my addr 172.21.15.81:0/3143698920 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3400640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 0x7f08e414e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e8e00640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e4154780 0x7f08e4151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 -- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 msgr2=0x7f08e414e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 0x7f08e414e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 -- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e4154780 msgr2=0x7f08e4151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e4154780 0x7f08e4151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 -- 172.21.15.81:0/3143698920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e414ea50 con 0x7f08e4072a60 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3400640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 0x7f08e414e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e3e00640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c390 tx=0x7f08d400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40074e0 con 0x7f08e4072a60 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08d40047f0 con 0x7f08e4072a60 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40076a0 con 0x7f08e4072a60 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41f8c00 con 0x7f08e4072a60 2024-05-27T06:59:41.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41f91c0 con 0x7f08e4072a60 2024-05-27T06:59:41.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e4050870 con 0x7f08e4072a60 2024-05-27T06:59:41.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.560+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f08d401a020 con 0x7f08e4072a60 2024-05-27T06:59:41.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.564+0000 7f08dbe00640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 0x7f08c4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:41.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.564+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f08d408ebe0 con 0x7f08e4072a60 2024-05-27T06:59:41.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.564+0000 7f08e3400640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 0x7f08c4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:41.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.564+0000 7f08e3400640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 0x7f08c4071940 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f08e414d7f0 tx=0x7f08d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:41.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.568+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f08d405b840 con 0x7f08e4072a60 2024-05-27T06:59:41.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.760+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08e40cbd60 con 0x7f08e4072a60 2024-05-27T06:59:41.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08dbe00640 1 -- 172.21.15.81:0/3143698920 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f08d405b140 con 0x7f08e4072a60 2024-05-27T06:59:41.755 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:41.755 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:41.756 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:41.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 msgr2=0x7f08c4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 0x7f08c4071940 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f08e414d7f0 tx=0x7f08d0008040 comp rx=0 tx=0).stop 2024-05-27T06:59:41.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 msgr2=0x7f08e406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c390 tx=0x7f08d400c850 comp rx=0 tx=0).stop 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 shutdown_connections 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f08e40c9630 0x7f08e414e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f08c406f430 0x7f08c4071940 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08e4154780 0x7f08e4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 --2- 172.21.15.81:0/3143698920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08e4072a60 0x7f08e406ec30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 >> 172.21.15.81:0/3143698920 conn(0x7f08e4068880 msgr2=0x7f08e4164780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 shutdown_connections 2024-05-27T06:59:41.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:41.764+0000 7f08e9800640 1 -- 172.21.15.81:0/3143698920 wait complete. 2024-05-27T06:59:41.778 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:59:43.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:43.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.944+0000 7f8cb0800640 1 Processor -- start 2024-05-27T06:59:43.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.944+0000 7f8cb0800640 1 -- start start 2024-05-27T06:59:43.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cb0800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca81546f0 0x7f8ca8072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cb0800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8ca815a510 0x7f8ca815a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cb0800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca815db00 con 0x7f8ca80732f0 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cb0800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca815e6f0 con 0x7f8ca815a510 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cb0800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca80755b0 con 0x7f8ca81546f0 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8ca815a510 0x7f8ca815a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8ca815a510 0x7f8ca815a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34228/0 (socket says 172.21.15.81:34228) 2024-05-27T06:59:43.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 -- 172.21.15.81:0/1501792677 learned_addr learned my addr 172.21.15.81:0/1501792677 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cade00640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca81546f0 0x7f8ca8072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 -- 172.21.15.81:0/1501792677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca81546f0 msgr2=0x7f8ca8072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca81546f0 0x7f8ca8072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 -- 172.21.15.81:0/1501792677 >> v1:172.21.15.106:6789/0 conn(0x7f8ca80732f0 legacy=0x7f8ca80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:43.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.948+0000 7f8cae800640 1 -- 172.21.15.81:0/1501792677 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca815ae60 con 0x7f8ca815a510 2024-05-27T06:59:43.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cae800640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca815a510 0x7f8ca815a920 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009e50 tx=0x7f8ca401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bab41dd62362308 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:43.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8caca00640 1 -- 172.21.15.81:0/1501792677 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40b4070 con 0x7f8ca815a510 2024-05-27T06:59:43.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8caca00640 1 -- 172.21.15.81:0/1501792677 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca40af400 con 0x7f8ca815a510 2024-05-27T06:59:43.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8caca00640 1 -- 172.21.15.81:0/1501792677 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40af7d0 con 0x7f8ca815a510 2024-05-27T06:59:43.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 -- 172.21.15.81:0/1501792677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca815a510 msgr2=0x7f8ca815a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:43.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca815a510 0x7f8ca815a920 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009e50 tx=0x7f8ca401f210 comp rx=0 tx=0).stop 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 -- 172.21.15.81:0/1501792677 shutdown_connections 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca815a510 0x7f8ca815a920 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 --2- 172.21.15.81:0/1501792677 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca81546f0 0x7f8ca8072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 -- 172.21.15.81:0/1501792677 >> 172.21.15.81:0/1501792677 conn(0x7f8ca8070700 msgr2=0x7f8ca80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 -- 172.21.15.81:0/1501792677 shutdown_connections 2024-05-27T06:59:43.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.952+0000 7f8cb0800640 1 -- 172.21.15.81:0/1501792677 wait complete. 2024-05-27T06:59:43.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 Processor -- start 2024-05-27T06:59:43.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 -- start start 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ca81546f0 0x7f8ca815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca815a510 0x7f8ca81f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca815e0a0 con 0x7f8ca80732f0 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca815ddc0 con 0x7f8ca81546f0 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cb0800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca815e380 con 0x7f8ca815a510 2024-05-27T06:59:43.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cade00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cade00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34244/0 (socket says 172.21.15.81:34244) 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cade00640 1 -- 172.21.15.81:0/2808716414 learned_addr learned my addr 172.21.15.81:0/2808716414 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cad400640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ca81546f0 0x7f8ca815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.956+0000 7f8cae800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca815a510 0x7f8ca81f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 -- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca815a510 msgr2=0x7f8ca81f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca815a510 0x7f8ca81f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 -- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ca81546f0 msgr2=0x7f8ca815b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ca81546f0 0x7f8ca815b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 -- 172.21.15.81:0/2808716414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca81f3610 con 0x7f8ca80732f0 2024-05-27T06:59:43.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cade00640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00d620 tx=0x7f8c9c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c00b620 con 0x7f8ca80732f0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9c019070 con 0x7f8ca80732f0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c00b9e0 con 0x7f8ca80732f0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca81f71e0 con 0x7f8ca80732f0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca81f68b0 con 0x7f8ca80732f0 2024-05-27T06:59:43.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca806a560 con 0x7f8ca80732f0 2024-05-27T06:59:43.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8c9c00bb80 con 0x7f8ca80732f0 2024-05-27T06:59:43.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 0x7f8c8c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:43.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.960+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f8c9c08f8f0 con 0x7f8ca80732f0 2024-05-27T06:59:43.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.964+0000 7f8cad400640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 0x7f8c8c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:43.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.964+0000 7f8cad400640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 0x7f8c8c071910 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8ca814e8c0 tx=0x7f8c98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:43.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:43.968+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8c9c05bf10 con 0x7f8ca80732f0 2024-05-27T06:59:44.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.156+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ca80c8e20 con 0x7f8ca80732f0 2024-05-27T06:59:44.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.156+0000 7f8ca2000640 1 -- 172.21.15.81:0/2808716414 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f8c9c060210 con 0x7f8ca80732f0 2024-05-27T06:59:44.149 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:44.150 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:44.150 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 msgr2=0x7f8c8c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 0x7f8c8c071910 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8ca814e8c0 tx=0x7f8c98009040 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 msgr2=0x7f8ca80cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00d620 tx=0x7f8c9c00dae0 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 shutdown_connections 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8ca815a510 0x7f8ca81f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8c8c06f400 0x7f8c8c071910 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ca81546f0 0x7f8ca815b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 --2- 172.21.15.81:0/2808716414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ca80732f0 0x7f8ca80cda60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:44.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 >> 172.21.15.81:0/2808716414 conn(0x7f8ca8070700 msgr2=0x7f8ca8151230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:44.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 shutdown_connections 2024-05-27T06:59:44.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:44.160+0000 7f8cb0800640 1 -- 172.21.15.81:0/2808716414 wait complete. 2024-05-27T06:59:44.171 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:45.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~222s 2024-05-27T06:59:46.173 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 Processor -- start 2024-05-27T06:59:46.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 -- start start 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 0x7f8090152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f809006c790 0x7f809006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 0x7f80900c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f80900c58b0 con 0x7f809006c790 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f80900c83f0 con 0x7f809015cb80 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8096200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f80900c9f80 con 0x7f80900c8880 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f809006c790 0x7f809006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f809006c790 0x7f809006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47028/0 (socket says 172.21.15.81:47028) 2024-05-27T06:59:46.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 -- 172.21.15.81:0/1165556415 learned_addr learned my addr 172.21.15.81:0/1165556415 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f808f400640 1 --2- 172.21.15.81:0/1165556415 >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 0x7f8090152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f808fe00640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 0x7f80900c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 -- 172.21.15.81:0/1165556415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 msgr2=0x7f80900c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 0x7f80900c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 -- 172.21.15.81:0/1165556415 >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 msgr2=0x7f8090152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 --2- 172.21.15.81:0/1165556415 >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 0x7f8090152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f8087e00640 1 -- 172.21.15.81:0/1165556415 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80900c52f0 con 0x7f809006c790 2024-05-27T06:59:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f808f400640 1 --2- 172.21.15.81:0/1165556415 >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 0x7f8090152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:59:46.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.340+0000 7f808fe00640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 0x7f80900c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T06:59:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8087e00640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f809006cba0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f809006d3d0 tx=0x7f807801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703f46f1d39a1356 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f808ea00640 1 -- 172.21.15.81:0/1165556415 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780b0070 con 0x7f809006c790 2024-05-27T06:59:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f808ea00640 1 -- 172.21.15.81:0/1165556415 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80780ab600 con 0x7f809006c790 2024-05-27T06:59:46.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f808ea00640 1 -- 172.21.15.81:0/1165556415 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780ab9d0 con 0x7f809006c790 2024-05-27T06:59:46.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 -- 172.21.15.81:0/1165556415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 msgr2=0x7f809006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:46.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f809006cba0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f809006d3d0 tx=0x7f807801c520 comp rx=0 tx=0).stop 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 -- 172.21.15.81:0/1165556415 shutdown_connections 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f80900c8880 0x7f80900c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 --2- 172.21.15.81:0/1165556415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f809006cba0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 --2- 172.21.15.81:0/1165556415 >> v2:172.21.15.106:3300/0 conn(0x7f809015cb80 0x7f8090152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 -- 172.21.15.81:0/1165556415 >> 172.21.15.81:0/1165556415 conn(0x7f8090070700 msgr2=0x7f80900cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 -- 172.21.15.81:0/1165556415 shutdown_connections 2024-05-27T06:59:46.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.344+0000 7f8096200640 1 -- 172.21.15.81:0/1165556415 wait complete. 2024-05-27T06:59:46.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 Processor -- start 2024-05-27T06:59:46.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 -- start start 2024-05-27T06:59:46.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80900c8880 0x7f809014b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809015cb80 0x7f8090151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8090153370 con 0x7f809006c790 2024-05-27T06:59:46.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f80901531f0 con 0x7f80900c8880 2024-05-27T06:59:46.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8096200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f80900c9b00 con 0x7f809015cb80 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47036/0 (socket says 172.21.15.81:47036) 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 -- 172.21.15.81:0/2107437839 learned_addr learned my addr 172.21.15.81:0/2107437839 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808fe00640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809015cb80 0x7f8090151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f8087e00640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80900c8880 0x7f809014b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 -- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809015cb80 msgr2=0x7f8090151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809015cb80 0x7f8090151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 -- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80900c8880 msgr2=0x7f809014b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80900c8880 0x7f809014b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.348+0000 7f808f400640 1 -- 172.21.15.81:0/2107437839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8090155980 con 0x7f809006c790 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f808f400640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f808000c820 tx=0x7f808000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:46.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080015070 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8080007a30 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809014c510 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8080007e40 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80901f9680 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809006a560 con 0x7f809006c790 2024-05-27T06:59:46.345 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8080018070 con 0x7f809006c790 2024-05-27T06:59:46.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 0x7f806c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f808008f3c0 con 0x7f809006c790 2024-05-27T06:59:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.352+0000 7f8087e00640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 0x7f806c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.356+0000 7f8087e00640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 0x7f806c071940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f80780ad000 tx=0x7f8078008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:46.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.360+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f808005c0b0 con 0x7f809006c790 2024-05-27T06:59:46.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.548+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8090072a00 con 0x7f809006c790 2024-05-27T06:59:46.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.548+0000 7f8087400640 1 -- 172.21.15.81:0/2107437839 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f808005b9b0 con 0x7f809006c790 2024-05-27T06:59:46.542 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:46.542 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:46.542 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 msgr2=0x7f806c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 0x7f806c071940 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f80780ad000 tx=0x7f8078008040 comp rx=0 tx=0).stop 2024-05-27T06:59:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 msgr2=0x7f8090155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f808000c820 tx=0x7f808000cce0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 shutdown_connections 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f809015cb80 0x7f8090151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f806c06f430 0x7f806c071940 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f80900c8880 0x7f809014b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 --2- 172.21.15.81:0/2107437839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809006c790 0x7f8090155440 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 >> 172.21.15.81:0/2107437839 conn(0x7f8090070700 msgr2=0x7f80900cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 shutdown_connections 2024-05-27T06:59:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:46.552+0000 7f8096200640 1 -- 172.21.15.81:0/2107437839 wait complete. 2024-05-27T06:59:46.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:48.565 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:48.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 Processor -- start 2024-05-27T06:59:48.725 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 -- start start 2024-05-27T06:59:48.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6df00c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df015db00 con 0x7f6df01546f0 2024-05-27T06:59:48.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df0073860 con 0x7f6df00c6fd0 2024-05-27T06:59:48.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.732+0000 7f6df5600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6df015e6f0 con 0x7f6df015ac20 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6deea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6df01546f0 0x7f6df0072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56612/0 (socket says 172.21.15.81:56612) 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6deea00640 1 -- 172.21.15.81:0/1410726528 learned_addr learned my addr 172.21.15.81:0/1410726528 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6df00c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559246343 0 0) 0x7f6df015db00 con 0x7f6df01546f0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd4003530 con 0x7f6df01546f0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104532158 0 0) 0x7f6df0073860 con 0x7f6df00c6fd0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6df015db00 con 0x7f6df00c6fd0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 127319574 0 0) 0x7f6dd4003530 con 0x7f6df01546f0 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 msgr2=0x7f6df00c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6df00c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 >> v1:172.21.15.106:6789/0 conn(0x7f6df00c6fd0 legacy=0x7f6df015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df015ddc0 con 0x7f6df01546f0 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f6de0001700 con 0x7f6df01546f0 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 >> v1:172.21.15.81:6789/0 conn(0x7f6df01546f0 legacy=0x7f6df0072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dd4003c20 0x7f6dd4004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6dd4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd4004da0 0x7f6dd4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.730 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6df015e6f0 con 0x7f6dd4004da0 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd400bbe0 con 0x7f6dd4003c20 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd400bde0 con 0x7f6df015ac20 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd4004da0 0x7f6dd4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6deea00640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dd4003c20 0x7f6dd4004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6de6000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6dd4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 -- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 msgr2=0x7f6dd4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6dd4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 -- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dd4003c20 msgr2=0x7f6dd4004070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6dd4003c20 0x7f6dd4004070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6def400640 1 -- 172.21.15.81:0/1410726528 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd4003530 con 0x7f6dd4004da0 2024-05-27T06:59:48.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.736+0000 7f6de6000640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df015ac20 0x7f6dd4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6def400640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd4004da0 0x7f6dd4005230 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009f90 tx=0x7f6de801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14d9abcd69fe2a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de80b2070 con 0x7f6dd4004da0 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de80a5c70 con 0x7f6dd4004da0 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6dee000640 1 -- 172.21.15.81:0/1410726528 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de8009c10 con 0x7f6dd4004da0 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- 172.21.15.81:0/1410726528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd4004da0 msgr2=0x7f6dd4005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 --2- 172.21.15.81:0/1410726528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd4004da0 0x7f6dd4005230 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6de8009f90 tx=0x7f6de801cc60 comp rx=0 tx=0).stop 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6def400640 1 -- 172.21.15.81:0/1410726528 reap_dead start 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- 172.21.15.81:0/1410726528 shutdown_connections 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- 172.21.15.81:0/1410726528 >> 172.21.15.81:0/1410726528 conn(0x7f6df0070700 msgr2=0x7f6df00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- 172.21.15.81:0/1410726528 shutdown_connections 2024-05-27T06:59:48.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- 172.21.15.81:0/1410726528 wait complete. 2024-05-27T06:59:48.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 Processor -- start 2024-05-27T06:59:48.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.740+0000 7f6df5600640 1 -- start start 2024-05-27T06:59:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df00c6b60 0x7f6df0150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6df0148ef0 0x7f6df0154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6df015e380 con 0x7f6df00c93c0 2024-05-27T06:59:48.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6df00755b0 con 0x7f6df0148ef0 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6df0154b00 con 0x7f6df00c6b60 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47064/0 (socket says 172.21.15.81:47064) 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 -- 172.21.15.81:0/2448230262 learned_addr learned my addr 172.21.15.81:0/2448230262 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6deea00640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df00c6b60 0x7f6df0150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6def400640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6df0148ef0 0x7f6df0154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 -- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df00c6b60 msgr2=0x7f6df0150300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df00c6b60 0x7f6df0150300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 -- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6df0148ef0 msgr2=0x7f6df0154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6df0148ef0 0x7f6df0154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 -- 172.21.15.81:0/2448230262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df0149300 con 0x7f6df00c93c0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de6000640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800e780 tx=0x7f6dd800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80050d0 con 0x7f6df00c93c0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd8005270 con 0x7f6df00c93c0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df00c9040 con 0x7f6df00c93c0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8005440 con 0x7f6df00c93c0 2024-05-27T06:59:48.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df00ca6c0 con 0x7f6df00c93c0 2024-05-27T06:59:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.744+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df006a560 con 0x7f6df00c93c0 2024-05-27T06:59:48.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.748+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6dd801a020 con 0x7f6df00c93c0 2024-05-27T06:59:48.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.748+0000 7f6de7400640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 0x7f6dcc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:48.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.748+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6dd808ea80 con 0x7f6df00c93c0 2024-05-27T06:59:48.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.748+0000 7f6deea00640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 0x7f6dcc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:48.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.748+0000 7f6deea00640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 0x7f6dcc071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6de0004850 tx=0x7f6de0000ff0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:48.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.752+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6dd8090e20 con 0x7f6df00c93c0 2024-05-27T06:59:48.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.940+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6df00c8e20 con 0x7f6df00c93c0 2024-05-27T06:59:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.940+0000 7f6de7400640 1 -- 172.21.15.81:0/2448230262 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6dd805b770 con 0x7f6df00c93c0 2024-05-27T06:59:48.933 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:48.933 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:48.934 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:48.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 msgr2=0x7f6dcc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 0x7f6dcc071940 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6de0004850 tx=0x7f6de0000ff0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 msgr2=0x7f6df01489b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800e780 tx=0x7f6dd800eb80 comp rx=0 tx=0).stop 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 shutdown_connections 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6dcc06f430 0x7f6dcc071940 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6df0148ef0 0x7f6df0154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6df00c93c0 0x7f6df01489b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 --2- 172.21.15.81:0/2448230262 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6df00c6b60 0x7f6df0150300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 >> 172.21.15.81:0/2448230262 conn(0x7f6df0070700 msgr2=0x7f6df015a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 shutdown_connections 2024-05-27T06:59:48.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:48.944+0000 7f6df5600640 1 -- 172.21.15.81:0/2448230262 wait complete. 2024-05-27T06:59:48.955 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:50.956 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:51.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.120+0000 7f560a200640 1 Processor -- start 2024-05-27T06:59:51.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.120+0000 7f560a200640 1 -- start start 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f560a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f560415cb80 0x7f5604152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f560a200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f56040c8f90 0x7f560406cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f560a200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56040c9f80 con 0x7f560406c790 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f560a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f56040c58b0 con 0x7f560415cb80 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f560a200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f56040c83f0 con 0x7f56040c8f90 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f560415cb80 0x7f5604152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f560415cb80 0x7f5604152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47074/0 (socket says 172.21.15.81:47074) 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 -- 172.21.15.81:0/1141681618 learned_addr learned my addr 172.21.15.81:0/1141681618 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:51.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603e00640 1 --2- 172.21.15.81:0/1141681618 >> v2:172.21.15.106:3300/0 conn(0x7f56040c8f90 0x7f560406cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 -- 172.21.15.81:0/1141681618 >> v2:172.21.15.106:3300/0 conn(0x7f56040c8f90 msgr2=0x7f560406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 --2- 172.21.15.81:0/1141681618 >> v2:172.21.15.106:3300/0 conn(0x7f56040c8f90 0x7f560406cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 -- 172.21.15.81:0/1141681618 >> v1:172.21.15.106:6789/0 conn(0x7f560406c790 legacy=0x7f56040c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.124+0000 7f5603400640 1 -- 172.21.15.81:0/1141681618 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56040c52f0 con 0x7f560415cb80 2024-05-27T06:59:51.119 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f5603400640 1 --2- 172.21.15.81:0/1141681618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f5604152cb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f56040ce960 tx=0x7f55f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da920fccec51e8f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f5602a00640 1 -- 172.21.15.81:0/1141681618 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f40b0070 con 0x7f560415cb80 2024-05-27T06:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f5602a00640 1 -- 172.21.15.81:0/1141681618 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f40ab600 con 0x7f560415cb80 2024-05-27T06:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f5602a00640 1 -- 172.21.15.81:0/1141681618 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f40ab9d0 con 0x7f560415cb80 2024-05-27T06:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 -- 172.21.15.81:0/1141681618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 msgr2=0x7f5604152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 --2- 172.21.15.81:0/1141681618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f5604152cb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f56040ce960 tx=0x7f55f401c520 comp rx=0 tx=0).stop 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 -- 172.21.15.81:0/1141681618 shutdown_connections 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 --2- 172.21.15.81:0/1141681618 >> v2:172.21.15.106:3300/0 conn(0x7f56040c8f90 0x7f560406cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 --2- 172.21.15.81:0/1141681618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f5604152cb0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 -- 172.21.15.81:0/1141681618 >> 172.21.15.81:0/1141681618 conn(0x7f5604070700 msgr2=0x7f56040cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 -- 172.21.15.81:0/1141681618 shutdown_connections 2024-05-27T06:59:51.121 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 -- 172.21.15.81:0/1141681618 wait complete. 2024-05-27T06:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.128+0000 7f560a200640 1 Processor -- start 2024-05-27T06:59:51.123 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- start start 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f560406c790 0x7f56041f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56040c8f90 0x7f560413e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56040c9c80 con 0x7f560415cb80 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f56040c9b00 con 0x7f56040c8f90 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f56041634e0 con 0x7f560406c790 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47086/0 (socket says 172.21.15.81:47086) 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 -- 172.21.15.81:0/1124851070 learned_addr learned my addr 172.21.15.81:0/1124851070 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:51.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f55faa00640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56040c8f90 0x7f560413e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603400640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f560406c790 0x7f56041f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 -- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f560406c790 msgr2=0x7f56041f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f560406c790 0x7f56041f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 -- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56040c8f90 msgr2=0x7f560413e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56040c8f90 0x7f560413e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 -- 172.21.15.81:0/1124851070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041f62e0 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f5603e00640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00efc0 tx=0x7f55fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc018020 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc014480 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc009e60 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56041f9310 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56041f98d0 con 0x7f560415cb80 2024-05-27T06:59:51.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.132+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560406a560 con 0x7f560415cb80 2024-05-27T06:59:51.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.136+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f55fc01b020 con 0x7f560415cb80 2024-05-27T06:59:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.136+0000 7f55fbe00640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 0x7f55e0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.136+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f55fc090120 con 0x7f560415cb80 2024-05-27T06:59:51.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.136+0000 7f5603400640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 0x7f55e0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:51.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.136+0000 7f5603400640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 0x7f55e0071910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f55f40ad000 tx=0x7f55f40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:51.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.140+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f55fc022030 con 0x7f560415cb80 2024-05-27T06:59:51.323 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56041f90a0 con 0x7f560415cb80 2024-05-27T06:59:51.324 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f55fbe00640 1 -- 172.21.15.81:0/1124851070 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f55fc010040 con 0x7f560415cb80 2024-05-27T06:59:51.324 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:51.324 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:51.325 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:51.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 msgr2=0x7f55e0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 0x7f55e0071910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f55f40ad000 tx=0x7f55f40a6000 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 msgr2=0x7f560413e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00efc0 tx=0x7f55fc00c490 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 shutdown_connections 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f55e006f400 0x7f55e0071910 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f560415cb80 0x7f560413e5d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f56040c8f90 0x7f560413e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 --2- 172.21.15.81:0/1124851070 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f560406c790 0x7f56041f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 >> 172.21.15.81:0/1124851070 conn(0x7f5604070700 msgr2=0x7f5604073df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 shutdown_connections 2024-05-27T06:59:51.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:51.332+0000 7f560a200640 1 -- 172.21.15.81:0/1124851070 wait complete. 2024-05-27T06:59:51.347 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:52.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~228s 2024-05-27T06:59:53.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:53.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 Processor -- start 2024-05-27T06:59:53.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 -- start start 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 0x7f3f300c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f30072840 con 0x7f3f30072430 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f300c5270 con 0x7f3f30161cd0 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f38800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3f300c5040 con 0x7f3f300c5b60 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f35e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3f30072430 0x7f3f30159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56618/0 (socket says 172.21.15.81:56618) 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f35e00640 1 -- 172.21.15.81:0/1954036352 learned_addr learned my addr 172.21.15.81:0/1954036352 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:53.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f35400640 1 --2- 172.21.15.81:0/1954036352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 0x7f3f300c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:53.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633326728 0 0) 0x7f3f30072840 con 0x7f3f30072430 2024-05-27T06:59:53.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f18003530 con 0x7f3f30072430 2024-05-27T06:59:53.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588781926 0 0) 0x7f3f300c5270 con 0x7f3f30161cd0 2024-05-27T06:59:53.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f30072840 con 0x7f3f30161cd0 2024-05-27T06:59:53.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f35400640 1 -- 172.21.15.81:0/1954036352 >> v1:172.21.15.106:6789/0 conn(0x7f3f30161cd0 legacy=0x7f3f301640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.508+0000 7f3f35400640 1 -- 172.21.15.81:0/1954036352 >> v1:172.21.15.81:6789/0 conn(0x7f3f30072430 legacy=0x7f3f30159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.512+0000 7f3f35400640 1 -- 172.21.15.81:0/1954036352 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f300c7810 con 0x7f3f300c5b60 2024-05-27T06:59:53.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.512+0000 7f3f35400640 1 --2- 172.21.15.81:0/1954036352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 0x7f3f300c5f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f200097f0 tx=0x7f3f2001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7081dc52f94b2cf server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:53.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.512+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f200b0070 con 0x7f3f300c5b60 2024-05-27T06:59:53.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.512+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f200ab600 con 0x7f3f300c5b60 2024-05-27T06:59:53.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.512+0000 7f3f34a00640 1 -- 172.21.15.81:0/1954036352 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f200ab9b0 con 0x7f3f300c5b60 2024-05-27T06:59:53.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- 172.21.15.81:0/1954036352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 msgr2=0x7f3f300c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 --2- 172.21.15.81:0/1954036352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 0x7f3f300c5f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f200097f0 tx=0x7f3f2001c4e0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- 172.21.15.81:0/1954036352 shutdown_connections 2024-05-27T06:59:53.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 --2- 172.21.15.81:0/1954036352 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f300c5b60 0x7f3f300c5f70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- 172.21.15.81:0/1954036352 >> 172.21.15.81:0/1954036352 conn(0x7f3f30070700 msgr2=0x7f3f300cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:53.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- 172.21.15.81:0/1954036352 shutdown_connections 2024-05-27T06:59:53.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- 172.21.15.81:0/1954036352 wait complete. 2024-05-27T06:59:53.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 Processor -- start 2024-05-27T06:59:53.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.516+0000 7f3f38800640 1 -- start start 2024-05-27T06:59:53.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f30072430 0x7f3f301573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:53.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f300c5b60 0x7f3f30142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300c7dd0 con 0x7f3f30161cd0 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300c75c0 con 0x7f3f300c5b60 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3f300c5610 con 0x7f3f30072430 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47098/0 (socket says 172.21.15.81:47098) 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 -- 172.21.15.81:0/2194961977 learned_addr learned my addr 172.21.15.81:0/2194961977 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f35400640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f300c5b60 0x7f3f30142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f35e00640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f30072430 0x7f3f301573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 -- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f30072430 msgr2=0x7f3f301573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f30072430 0x7f3f301573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 -- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f300c5b60 msgr2=0x7f3f30142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f300c5b60 0x7f3f30142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 -- 172.21.15.81:0/2194961977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f30142d40 con 0x7f3f30161cd0 2024-05-27T06:59:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f36800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c760 tx=0x7f3f2c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c003b80 con 0x7f3f30161cd0 2024-05-27T06:59:53.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c003d20 con 0x7f3f30161cd0 2024-05-27T06:59:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f300cd220 con 0x7f3f30161cd0 2024-05-27T06:59:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c00aa30 con 0x7f3f30161cd0 2024-05-27T06:59:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f30141d20 con 0x7f3f30161cd0 2024-05-27T06:59:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3006a560 con 0x7f3f30161cd0 2024-05-27T06:59:53.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.520+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3f2c00abd0 con 0x7f3f30161cd0 2024-05-27T06:59:53.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.524+0000 7f3f2a000640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 0x7f3f14071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:53.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.524+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f3f2c08f240 con 0x7f3f30161cd0 2024-05-27T06:59:53.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.524+0000 7f3f35e00640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 0x7f3f14071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:53.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.524+0000 7f3f35e00640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 0x7f3f14071910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000c10 tx=0x7f3f24007ef0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:53.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.528+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3f2c05b8e0 con 0x7f3f30161cd0 2024-05-27T06:59:53.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.720+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f30161910 con 0x7f3f30161cd0 2024-05-27T06:59:53.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.720+0000 7f3f2a000640 1 -- 172.21.15.81:0/2194961977 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f3f2c05fbe0 con 0x7f3f30161cd0 2024-05-27T06:59:53.714 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:53.714 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:53.714 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 msgr2=0x7f3f14071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 0x7f3f14071910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000c10 tx=0x7f3f24007ef0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 msgr2=0x7f3f301479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c00c760 tx=0x7f3f2c00cc20 comp rx=0 tx=0).stop 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 shutdown_connections 2024-05-27T06:59:53.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3f1406f400 0x7f3f14071910 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30161cd0 0x7f3f301479a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3f300c5b60 0x7f3f30142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 --2- 172.21.15.81:0/2194961977 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3f30072430 0x7f3f301573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 >> 172.21.15.81:0/2194961977 conn(0x7f3f30070700 msgr2=0x7f3f300761a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 shutdown_connections 2024-05-27T06:59:53.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:53.724+0000 7f3f38800640 1 -- 172.21.15.81:0/2194961977 wait complete. 2024-05-27T06:59:53.737 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T06:59:55.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:55.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 Processor -- start 2024-05-27T06:59:55.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 -- start start 2024-05-27T06:59:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea48072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fea480732f0 0x7fea480c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea4815db00 con 0x7fea480c7510 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fea4815e6f0 con 0x7fea480732f0 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea50600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fea480755b0 con 0x7fea481546f0 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea4e600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea480c7510 0x7fea4815a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46406/0 (socket says 172.21.15.81:46406) 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea4e600640 1 -- 172.21.15.81:0/3958201015 learned_addr learned my addr 172.21.15.81:0/3958201015 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea4dc00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea48072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.904+0000 7fea4d200640 1 --2- 172.21.15.81:0/3958201015 >> v2:172.21.15.106:3300/0 conn(0x7fea480732f0 0x7fea480c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647624383 0 0) 0x7fea4815db00 con 0x7fea480c7510 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea30003530 con 0x7fea480c7510 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1303117503 0 0) 0x7fea30003530 con 0x7fea480c7510 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 msgr2=0x7fea48072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea48072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 >> v2:172.21.15.106:3300/0 conn(0x7fea480732f0 msgr2=0x7fea480c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 --2- 172.21.15.81:0/3958201015 >> v2:172.21.15.106:3300/0 conn(0x7fea480732f0 0x7fea480c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea48073860 con 0x7fea480c7510 2024-05-27T06:59:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fea40001a80 con 0x7fea480c7510 2024-05-27T06:59:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 >> v1:172.21.15.81:6789/0 conn(0x7fea480c7510 legacy=0x7fea4815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea30003a10 0x7fea30003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea30004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30004d50 0x7fea30005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea4815e6f0 con 0x7fea30003a10 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea480755b0 con 0x7fea30004d50 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fea3000bb00 con 0x7fea481546f0 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea30003a10 0x7fea30003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4d200640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea30004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4e600640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30004d50 0x7fea30005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 -- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 msgr2=0x7fea30004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea481546f0 0x7fea30004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 -- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30004d50 msgr2=0x7fea30005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea30004d50 0x7fea30005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.908+0000 7fea4dc00640 1 -- 172.21.15.81:0/3958201015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea30003530 con 0x7fea30003a10 2024-05-27T06:59:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea4dc00640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea30003a10 0x7fea30003e80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ae0 tx=0x7fea3c01cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d44157c2caf064a server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0b2070 con 0x7fea30003a10 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3c0a5720 con 0x7fea30003a10 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea47e00640 1 -- 172.21.15.81:0/3958201015 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c0096b0 con 0x7fea30003a10 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- 172.21.15.81:0/3958201015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea30003a10 msgr2=0x7fea30003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 --2- 172.21.15.81:0/3958201015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea30003a10 0x7fea30003e80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ae0 tx=0x7fea3c01cb20 comp rx=0 tx=0).stop 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea4e600640 1 -- 172.21.15.81:0/3958201015 reap_dead start 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- 172.21.15.81:0/3958201015 shutdown_connections 2024-05-27T06:59:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- 172.21.15.81:0/3958201015 >> 172.21.15.81:0/3958201015 conn(0x7fea48070700 msgr2=0x7fea480cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- 172.21.15.81:0/3958201015 shutdown_connections 2024-05-27T06:59:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- 172.21.15.81:0/3958201015 wait complete. 2024-05-27T06:59:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 Processor -- start 2024-05-27T06:59:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.912+0000 7fea50600640 1 -- start start 2024-05-27T06:59:55.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 0x7fea4815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 0x7fea48151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea4815e380 con 0x7fea48157280 2024-05-27T06:59:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea4815e0a0 con 0x7fea48156e70 2024-05-27T06:59:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fea48154b00 con 0x7fea30003a10 2024-05-27T06:59:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56726/0 (socket says 172.21.15.81:56726) 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 -- 172.21.15.81:0/4166384578 learned_addr learned my addr 172.21.15.81:0/4166384578 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4dc00640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 0x7fea4815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4d200640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 0x7fea48151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 -- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 msgr2=0x7fea4815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 0x7fea4815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 -- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 msgr2=0x7fea48151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 0x7fea48151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 -- 172.21.15.81:0/4166384578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea481f3530 con 0x7fea48157280 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4dc00640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 0x7fea4815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4d200640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 0x7fea48151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea4e600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fea480725c0 tx=0x7fea40012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40007070 con 0x7fea48157280 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea40007230 con 0x7fea48157280 2024-05-27T06:59:55.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea48155010 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40016680 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea481555d0 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.916+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea48154e00 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.920+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fea4001e020 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.920+0000 7fea45600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 0x7fea28071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.920+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fea40092f10 con 0x7fea48157280 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.920+0000 7fea4dc00640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 0x7fea28071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.920+0000 7fea4dc00640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 0x7fea28071940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ae0 tx=0x7fea3c0046d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:55.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:55.924+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fea4005fb70 con 0x7fea48157280 2024-05-27T06:59:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.116+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea481f32c0 con 0x7fea48157280 2024-05-27T06:59:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.116+0000 7fea45600640 1 -- 172.21.15.81:0/4166384578 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fea4005f470 con 0x7fea48157280 2024-05-27T06:59:56.110 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:56.110 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 msgr2=0x7fea28071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 0x7fea28071940 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009ae0 tx=0x7fea3c0046d0 comp rx=0 tx=0).stop 2024-05-27T06:59:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 msgr2=0x7fea48152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fea480725c0 tx=0x7fea40012630 comp rx=0 tx=0).stop 2024-05-27T06:59:56.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 shutdown_connections 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fea2806f430 0x7fea28071940 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea48157280 0x7fea48152330 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea48156e70 0x7fea48151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 --2- 172.21.15.81:0/4166384578 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fea30003a10 0x7fea4815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 >> 172.21.15.81:0/4166384578 conn(0x7fea48070700 msgr2=0x7fea481514d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 shutdown_connections 2024-05-27T06:59:56.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:56.120+0000 7fea50600640 1 -- 172.21.15.81:0/4166384578 wait complete. 2024-05-27T06:59:56.132 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:58.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T06:59:58.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.296+0000 7f1546000640 1 Processor -- start 2024-05-27T06:59:58.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.296+0000 7f1546000640 1 -- start start 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1546000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f154015fd20 0x7f1540156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1546000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f15400cb010 0x7f1540072c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1546000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1540160130 con 0x7f15400c89a0 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1546000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f15400c9070 con 0x7f15400cb010 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1546000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1540074890 con 0x7f154015fd20 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1536a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f15400c89a0 0x7f15400ca900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46412/0 (socket says 172.21.15.81:46412) 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f1536a00640 1 -- 172.21.15.81:0/3873455647 learned_addr learned my addr 172.21.15.81:0/3873455647 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T06:59:58.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153fe00640 1 --2- 172.21.15.81:0/3873455647 >> v2:172.21.15.81:3300/0 conn(0x7f15400cb010 0x7f1540072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153f400640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f154015fd20 0x7f1540156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153fe00640 1 -- 172.21.15.81:0/3873455647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f154015fd20 msgr2=0x7f1540156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153fe00640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f154015fd20 0x7f1540156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153fe00640 1 -- 172.21.15.81:0/3873455647 >> v1:172.21.15.81:6789/0 conn(0x7f15400c89a0 legacy=0x7f15400ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153fe00640 1 -- 172.21.15.81:0/3873455647 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15400c94c0 con 0x7f15400cb010 2024-05-27T06:59:58.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.300+0000 7f153ea00640 1 -- 172.21.15.81:0/3873455647 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3644775748 0 0) 0x7f15280014a0 con 0x7f15400c89a0 2024-05-27T06:59:58.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f153fe00640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15400cb010 0x7f1540072c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f15380097c0 tx=0x7f153801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715863d5fea51a5e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:58.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f153ea00640 1 -- 172.21.15.81:0/3873455647 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15380b0070 con 0x7f15400cb010 2024-05-27T06:59:58.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f153ea00640 1 -- 172.21.15.81:0/3873455647 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15380ab600 con 0x7f15400cb010 2024-05-27T06:59:58.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f153ea00640 1 -- 172.21.15.81:0/3873455647 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15380ab9d0 con 0x7f15400cb010 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 -- 172.21.15.81:0/3873455647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15400cb010 msgr2=0x7f1540072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15400cb010 0x7f1540072c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f15380097c0 tx=0x7f153801c4a0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 -- 172.21.15.81:0/3873455647 shutdown_connections 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15400cb010 0x7f1540072c10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 --2- 172.21.15.81:0/3873455647 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f154015fd20 0x7f1540156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 -- 172.21.15.81:0/3873455647 >> 172.21.15.81:0/3873455647 conn(0x7f154006f230 msgr2=0x7f1540070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 -- 172.21.15.81:0/3873455647 shutdown_connections 2024-05-27T06:59:58.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.304+0000 7f1546000640 1 -- 172.21.15.81:0/3873455647 wait complete. 2024-05-27T06:59:58.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 Processor -- start 2024-05-27T06:59:58.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- start start 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 0x7f1540155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 0x7f1540154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15400c9760 con 0x7f1540154ea0 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1540073fe0 con 0x7f154015fd20 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f15401698d0 con 0x7f15400c89a0 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56744/0 (socket says 172.21.15.81:56744) 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 -- 172.21.15.81:0/4064735421 learned_addr learned my addr 172.21.15.81:0/4064735421 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153f400640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 0x7f1540155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1536a00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 0x7f1540154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 -- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 msgr2=0x7f1540155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 0x7f1540155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 -- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 msgr2=0x7f1540154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 0x7f1540154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 -- 172.21.15.81:0/4064735421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1540166960 con 0x7f1540154ea0 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153f400640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 0x7f1540155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1536a00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 0x7f1540154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T06:59:58.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f153fe00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f15380ae040 tx=0x7f153801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15380b0050 con 0x7f1540154ea0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15380b1040 con 0x7f1540154ea0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154014a720 con 0x7f1540154ea0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538006e20 con 0x7f1540154ea0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.308+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15401f64a0 con 0x7f1540154ea0 2024-05-27T06:59:58.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1540157070 con 0x7f1540154ea0 2024-05-27T06:59:58.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f15380be070 con 0x7f1540154ea0 2024-05-27T06:59:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f1537e00640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 0x7f151c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T06:59:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f15381292f0 con 0x7f1540154ea0 2024-05-27T06:59:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f153f400640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 0x7f151c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T06:59:58.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.312+0000 7f153f400640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 0x7f151c071940 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1530009b10 tx=0x7f1530008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T06:59:58.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.316+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f15380a4e20 con 0x7f1540154ea0 2024-05-27T06:59:58.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.508+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f154006a560 con 0x7f1540154ea0 2024-05-27T06:59:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.508+0000 7f1537e00640 1 -- 172.21.15.81:0/4064735421 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f15380fae20 con 0x7f1540154ea0 2024-05-27T06:59:58.502 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T06:59:58.502 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T06:59:58.502 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T06:59:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 msgr2=0x7f151c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 0x7f151c071940 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1530009b10 tx=0x7f1530008040 comp rx=0 tx=0).stop 2024-05-27T06:59:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 msgr2=0x7f15401479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T06:59:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f15380ae040 tx=0x7f153801ace0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 shutdown_connections 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f151c06f430 0x7f151c071940 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1540154ea0 0x7f15401479a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f154015fd20 0x7f1540154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 --2- 172.21.15.81:0/4064735421 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f15400c89a0 0x7f1540155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 >> 172.21.15.81:0/4064735421 conn(0x7f154006f230 msgr2=0x7f15400cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 shutdown_connections 2024-05-27T06:59:58.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T06:59:58.512+0000 7f1546000640 1 -- 172.21.15.81:0/4064735421 wait complete. 2024-05-27T06:59:58.524 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T06:59:59.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~235s 2024-05-27T07:00:00.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:00.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.688+0000 7f73fea00640 1 Processor -- start 2024-05-27T07:00:00.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.688+0000 7f73fea00640 1 -- start start 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73f80c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f8072840 con 0x7f73f8072430 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fea00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f80c5270 con 0x7f73f8161cd0 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fea00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f73f80c5040 con 0x7f73f80c5b60 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f73f8072430 0x7f73f8159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46422/0 (socket says 172.21.15.81:46422) 2024-05-27T07:00:00.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7e00640 1 -- 172.21.15.81:0/4168406732 learned_addr learned my addr 172.21.15.81:0/4168406732 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253865806 0 0) 0x7f73f8072840 con 0x7f73f8072430 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73ef400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73f80c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73dc003530 con 0x7f73f8072430 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3850399194 0 0) 0x7f73dc003530 con 0x7f73f8072430 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 msgr2=0x7f73f80c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73f80c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 >> v1:172.21.15.106:6789/0 conn(0x7f73f8161cd0 legacy=0x7f73f81640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f80c7810 con 0x7f73f8072430 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f73e8001700 con 0x7f73f8072430 2024-05-27T07:00:00.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 >> v1:172.21.15.81:6789/0 conn(0x7f73f8072430 legacy=0x7f73f8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73dc003a40 0x7f73dc003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73dc004d50 0x7f73dc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc00bb60 con 0x7f73dc004d50 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc00bd10 con 0x7f73dc003a40 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f73dc00bf10 con 0x7f73f80c5b60 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73dc004d50 0x7f73dc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73f7e00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73dc003a40 0x7f73dc003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73ef400640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 -- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 msgr2=0x7f73dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73f80c5b60 0x7f73dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 -- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73dc003a40 msgr2=0x7f73dc003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73dc003a40 0x7f73dc003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.692+0000 7f73fca00640 1 -- 172.21.15.81:0/4168406732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc003530 con 0x7f73dc004d50 2024-05-27T07:00:00.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fca00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73dc004d50 0x7f73dc005200 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f73f80c9dd0 tx=0x7f73f001ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39857ad8ee734c07 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:00.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0001b70 con 0x7f73dc004d50 2024-05-27T07:00:00.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f0003980 con 0x7f73dc004d50 2024-05-27T07:00:00.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73f7400640 1 -- 172.21.15.81:0/4168406732 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0003d70 con 0x7f73dc004d50 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 -- 172.21.15.81:0/4168406732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73dc004d50 msgr2=0x7f73dc005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 --2- 172.21.15.81:0/4168406732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73dc004d50 0x7f73dc005200 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f73f80c9dd0 tx=0x7f73f001ee20 comp rx=0 tx=0).stop 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fca00640 1 -- 172.21.15.81:0/4168406732 reap_dead start 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 -- 172.21.15.81:0/4168406732 shutdown_connections 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 -- 172.21.15.81:0/4168406732 >> 172.21.15.81:0/4168406732 conn(0x7f73f8070700 msgr2=0x7f73f80cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 -- 172.21.15.81:0/4168406732 shutdown_connections 2024-05-27T07:00:00.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.696+0000 7f73fea00640 1 -- 172.21.15.81:0/4168406732 wait complete. 2024-05-27T07:00:00.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 Processor -- start 2024-05-27T07:00:00.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 -- start start 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73dc004d50 0x7f73f8141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73f80c4530 0x7f73f80ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73f80c7dd0 con 0x7f73f80c4940 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f73f80c75c0 con 0x7f73f80c4530 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f73f81595e0 con 0x7f73dc004d50 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56760/0 (socket says 172.21.15.81:56760) 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 -- 172.21.15.81:0/901141537 learned_addr learned my addr 172.21.15.81:0/901141537 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:00.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73f7e00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73dc004d50 0x7f73f8141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73ef400640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73f80c4530 0x7f73f80ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 -- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73dc004d50 msgr2=0x7f73f8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73dc004d50 0x7f73f8141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 -- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73f80c4530 msgr2=0x7f73f80ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73f80c4530 0x7f73f80ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 -- 172.21.15.81:0/901141537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f81423b0 con 0x7f73f80c4940 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fca00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f73f00253b0 tx=0x7f73dc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0001b50 con 0x7f73f80c4940 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f0006d70 con 0x7f73f80c4940 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f81416b0 con 0x7f73f80c4940 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.700+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0005580 con 0x7f73f80c4940 2024-05-27T07:00:00.696 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f8161aa0 con 0x7f73f80c4940 2024-05-27T07:00:00.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f806a560 con 0x7f73f80c4940 2024-05-27T07:00:00.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f73f00bf060 con 0x7f73f80c4940 2024-05-27T07:00:00.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73f4c00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 0x7f73d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:00.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f73f012bd10 con 0x7f73f80c4940 2024-05-27T07:00:00.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.704+0000 7f73f7e00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 0x7f73d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:00.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.708+0000 7f73f7e00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 0x7f73d4071910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f73f8070fe0 tx=0x7f73e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:00.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.712+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f73f0131020 con 0x7f73f80c4940 2024-05-27T07:00:00.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.904+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73f80ce460 con 0x7f73f80c4940 2024-05-27T07:00:00.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73f4c00640 1 -- 172.21.15.81:0/901141537 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f73f0018b80 con 0x7f73f80c4940 2024-05-27T07:00:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:00.899 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:00.900 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:00.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 msgr2=0x7f73d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 0x7f73d4071910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f73f8070fe0 tx=0x7f73e8009040 comp rx=0 tx=0).stop 2024-05-27T07:00:00.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 msgr2=0x7f73f813c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:00.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f73f00253b0 tx=0x7f73dc005840 comp rx=0 tx=0).stop 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 shutdown_connections 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f73d406f400 0x7f73d4071910 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f80c4940 0x7f73f813c0e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f73f80c4530 0x7f73f80ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 --2- 172.21.15.81:0/901141537 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f73dc004d50 0x7f73f8141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 >> 172.21.15.81:0/901141537 conn(0x7f73f8070700 msgr2=0x7f73f8074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 shutdown_connections 2024-05-27T07:00:00.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:00.908+0000 7f73fea00640 1 -- 172.21.15.81:0/901141537 wait complete. 2024-05-27T07:00:00.921 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:02.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:03.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 Processor -- start 2024-05-27T07:00:03.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 -- start start 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f9250073140 0x7f92500c7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9250073910 con 0x7f92501549b0 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9250073630 con 0x7f92500c6d00 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f92500c9ba0 con 0x7f9250073140 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9255800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f92501549b0 0x7f925015dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46426/0 (socket says 172.21.15.81:46426) 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9255800640 1 -- 172.21.15.81:0/2055145165 learned_addr learned my addr 172.21.15.81:0/2055145165 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:03.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256200640 1 --2- 172.21.15.81:0/2055145165 >> v2:172.21.15.106:3300/0 conn(0x7f9250073140 0x7f92500c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:03.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644473585 0 0) 0x7f9244001120 con 0x7f92501549b0 2024-05-27T07:00:03.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9238003530 con 0x7f92501549b0 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566611796 0 0) 0x7f9250073630 con 0x7f92500c6d00 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9244001120 con 0x7f92500c6d00 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3120644634 0 0) 0x7f9238003530 con 0x7f92501549b0 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256200640 1 -- 172.21.15.81:0/2055145165 >> v1:172.21.15.106:6789/0 conn(0x7f92500c6d00 legacy=0x7f9250072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256200640 1 -- 172.21.15.81:0/2055145165 >> v1:172.21.15.81:6789/0 conn(0x7f92501549b0 legacy=0x7f925015dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.092+0000 7f9256200640 1 -- 172.21.15.81:0/2055145165 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92500c9270 con 0x7f9250073140 2024-05-27T07:00:03.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256200640 1 --2- 172.21.15.81:0/2055145165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9250073140 0x7f92500c7110 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f92500c9d20 tx=0x7f924801c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db82dcf56ae8f11e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:03.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440018d0 con 0x7f9250073140 2024-05-27T07:00:03.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480a64c0 con 0x7f9250073140 2024-05-27T07:00:03.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f924fe00640 1 -- 172.21.15.81:0/2055145165 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480a6870 con 0x7f9250073140 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 -- 172.21.15.81:0/2055145165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9250073140 msgr2=0x7f92500c7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 --2- 172.21.15.81:0/2055145165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9250073140 0x7f92500c7110 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f92500c9d20 tx=0x7f924801c870 comp rx=0 tx=0).stop 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 -- 172.21.15.81:0/2055145165 shutdown_connections 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 --2- 172.21.15.81:0/2055145165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9250073140 0x7f92500c7110 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 -- 172.21.15.81:0/2055145165 >> 172.21.15.81:0/2055145165 conn(0x7f925006d180 msgr2=0x7f92500cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 -- 172.21.15.81:0/2055145165 shutdown_connections 2024-05-27T07:00:03.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.096+0000 7f9256c00640 1 -- 172.21.15.81:0/2055145165 wait complete. 2024-05-27T07:00:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 Processor -- start 2024-05-27T07:00:03.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 -- start start 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92500c6d00 0x7f925014e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92501549b0 0x7f92500cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9250168950 con 0x7f9250073140 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92501687d0 con 0x7f92501549b0 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f9250168ad0 con 0x7f92500c6d00 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56776/0 (socket says 172.21.15.81:56776) 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 -- 172.21.15.81:0/575853043 learned_addr learned my addr 172.21.15.81:0/575853043 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9256200640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92501549b0 0x7f92500cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:03.094 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9254e00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92500c6d00 0x7f925014e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:03.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 -- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92500c6d00 msgr2=0x7f925014e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92500c6d00 0x7f925014e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 -- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92501549b0 msgr2=0x7f92500cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92501549b0 0x7f92500cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 -- 172.21.15.81:0/575853043 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250159620 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.100+0000 7f9255800640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f924400c700 tx=0x7f924400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244017070 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9244005cf0 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244007290 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9250159900 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92501f9770 con 0x7f9250073140 2024-05-27T07:00:03.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9250050870 con 0x7f9250073140 2024-05-27T07:00:03.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f924401a070 con 0x7f9250073140 2024-05-27T07:00:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 0x7f9230071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f924408e9d0 con 0x7f9250073140 2024-05-27T07:00:03.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.104+0000 7f9254e00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 0x7f9230071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:03.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.108+0000 7f9254e00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 0x7f9230071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f92501f7050 tx=0x7f923c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:03.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.108+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f9250073910 con 0x7f9250073140 2024-05-27T07:00:03.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.300+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92500c8c80 con 0x7f9250073140 2024-05-27T07:00:03.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.300+0000 7f924d600640 1 -- 172.21.15.81:0/575853043 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f924405b110 con 0x7f9250073140 2024-05-27T07:00:03.293 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:03.293 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:03.293 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:03.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 msgr2=0x7f9230071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 0x7f9230071910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f92501f7050 tx=0x7f923c009040 comp rx=0 tx=0).stop 2024-05-27T07:00:03.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 msgr2=0x7f92501f9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:03.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f924400c700 tx=0x7f924400cbc0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 shutdown_connections 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f92500c6d00 0x7f925014e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f923006f400 0x7f9230071910 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92501549b0 0x7f92500cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 --2- 172.21.15.81:0/575853043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9250073140 0x7f92501f9f70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 >> 172.21.15.81:0/575853043 conn(0x7f925006d180 msgr2=0x7f925014fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 shutdown_connections 2024-05-27T07:00:03.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:03.304+0000 7f9256c00640 1 -- 172.21.15.81:0/575853043 wait complete. 2024-05-27T07:00:03.316 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:05.116 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:00:05.120 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:00:05.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:05.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.484+0000 7f190fc00640 1 Processor -- start 2024-05-27T07:00:05.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.484+0000 7f190fc00640 1 -- start start 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.484+0000 7f190fc00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 0x7f1908159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190fc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f190815a040 0x7f19080c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190fc00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1908072840 con 0x7f19080c60a0 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190fc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f19080c5040 con 0x7f190815a040 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190fc00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f19080c75c0 con 0x7f1908072430 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f190815a040 0x7f19080c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f190815a040 0x7f19080c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43386/0 (socket says 172.21.15.81:43386) 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 -- 172.21.15.81:0/1485889900 learned_addr learned my addr 172.21.15.81:0/1485889900 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190d200640 1 --2- 172.21.15.81:0/1485889900 >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 0x7f1908159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 -- 172.21.15.81:0/1485889900 >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 msgr2=0x7f1908159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 --2- 172.21.15.81:0/1485889900 >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 0x7f1908159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 -- 172.21.15.81:0/1485889900 >> v1:172.21.15.106:6789/0 conn(0x7f19080c60a0 legacy=0x7f190815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 -- 172.21.15.81:0/1485889900 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19080c5270 con 0x7f190815a040 2024-05-27T07:00:05.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f190d200640 1 --2- 172.21.15.81:0/1485889900 >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 0x7f1908159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:05.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.488+0000 7f18ffe00640 1 --2- 172.21.15.81:0/1485889900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f190815a040 0x7f19080c5b60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1908071c00 tx=0x7f18f001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693ec1370904b319 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f1907e00640 1 -- 172.21.15.81:0/1485889900 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f00b0070 con 0x7f190815a040 2024-05-27T07:00:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f1907e00640 1 -- 172.21.15.81:0/1485889900 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f00ab600 con 0x7f190815a040 2024-05-27T07:00:05.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f1907e00640 1 -- 172.21.15.81:0/1485889900 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f00ab9d0 con 0x7f190815a040 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- 172.21.15.81:0/1485889900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f190815a040 msgr2=0x7f19080c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 --2- 172.21.15.81:0/1485889900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f190815a040 0x7f19080c5b60 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1908071c00 tx=0x7f18f001c520 comp rx=0 tx=0).stop 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- 172.21.15.81:0/1485889900 shutdown_connections 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 --2- 172.21.15.81:0/1485889900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f190815a040 0x7f19080c5b60 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 --2- 172.21.15.81:0/1485889900 >> v2:172.21.15.106:3300/0 conn(0x7f1908072430 0x7f1908159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- 172.21.15.81:0/1485889900 >> 172.21.15.81:0/1485889900 conn(0x7f1908070700 msgr2=0x7f19080cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- 172.21.15.81:0/1485889900 shutdown_connections 2024-05-27T07:00:05.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- 172.21.15.81:0/1485889900 wait complete. 2024-05-27T07:00:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 Processor -- start 2024-05-27T07:00:05.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.492+0000 7f190fc00640 1 -- start start 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19080c60a0 0x7f19081427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 0x7f190813f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19080ce6f0 con 0x7f1908072430 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f19080c7dd0 con 0x7f190815a040 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f19080c5610 con 0x7f19080c60a0 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43392/0 (socket says 172.21.15.81:43392) 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 -- 172.21.15.81:0/648037276 learned_addr learned my addr 172.21.15.81:0/648037276 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:05.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f18ffe00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19080c60a0 0x7f19081427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190dc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 0x7f190813f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 -- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19080c60a0 msgr2=0x7f19081427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19080c60a0 0x7f19081427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 -- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 msgr2=0x7f190813f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 0x7f190813f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 -- 172.21.15.81:0/648037276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190814b8c0 con 0x7f1908072430 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190dc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 0x7f190813f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190d200640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d830 tx=0x7f18f800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8018020 con 0x7f1908072430 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f80096a0 con 0x7f1908072430 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190813de80 con 0x7f1908072430 2024-05-27T07:00:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8009a90 con 0x7f1908072430 2024-05-27T07:00:05.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1908140bb0 con 0x7f1908072430 2024-05-27T07:00:05.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.496+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f190806a560 con 0x7f1908072430 2024-05-27T07:00:05.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.500+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f18f801b020 con 0x7f1908072430 2024-05-27T07:00:05.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.500+0000 7f1905600640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 0x7f18e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:05.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.500+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f18f808fa30 con 0x7f1908072430 2024-05-27T07:00:05.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.500+0000 7f18ffe00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 0x7f18e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:05.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.500+0000 7f18ffe00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 0x7f18e4071940 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18f00ad000 tx=0x7f18f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:05.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.504+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f18f8022020 con 0x7f1908072430 2024-05-27T07:00:05.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.696+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f190815cae0 con 0x7f1908072430 2024-05-27T07:00:05.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f1905600640 1 -- 172.21.15.81:0/648037276 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f18f805c690 con 0x7f1908072430 2024-05-27T07:00:05.691 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:05.691 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:05.692 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 msgr2=0x7f18e4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 0x7f18e4071940 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18f00ad000 tx=0x7f18f0008040 comp rx=0 tx=0).stop 2024-05-27T07:00:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 msgr2=0x7f190813e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:05.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f18f800d830 tx=0x7f18f800dcf0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 shutdown_connections 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f19080c60a0 0x7f19081427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f18e406f430 0x7f18e4071940 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f190815a040 0x7f190813f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 --2- 172.21.15.81:0/648037276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1908072430 0x7f190813e640 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 >> 172.21.15.81:0/648037276 conn(0x7f1908070700 msgr2=0x7f19080c45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 shutdown_connections 2024-05-27T07:00:05.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:05.700+0000 7f190fc00640 1 -- 172.21.15.81:0/648037276 wait complete. 2024-05-27T07:00:05.718 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:00:06.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~242s 2024-05-27T07:00:07.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:07.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.884+0000 7f0c82c00640 1 Processor -- start 2024-05-27T07:00:07.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.884+0000 7f0c82c00640 1 -- start start 2024-05-27T07:00:07.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c82c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0c7c0c5b60 0x7f0c7c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c82c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0c7c161cd0 0x7f0c7c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c82c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c7c072840 con 0x7f0c7c072430 2024-05-27T07:00:07.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c82c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c0c5040 con 0x7f0c7c161cd0 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c82c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c0c75c0 con 0x7f0c7c0c5b60 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0c7c161cd0 0x7f0c7c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0c7c161cd0 0x7f0c7c1640f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43408/0 (socket says 172.21.15.81:43408) 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 -- 172.21.15.81:0/377798269 learned_addr learned my addr 172.21.15.81:0/377798269 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c73e00640 1 --2- 172.21.15.81:0/377798269 >> v2:172.21.15.106:3300/0 conn(0x7f0c7c0c5b60 0x7f0c7c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 -- 172.21.15.81:0/377798269 >> v2:172.21.15.106:3300/0 conn(0x7f0c7c0c5b60 msgr2=0x7f0c7c0c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 --2- 172.21.15.81:0/377798269 >> v2:172.21.15.106:3300/0 conn(0x7f0c7c0c5b60 0x7f0c7c0c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 -- 172.21.15.81:0/377798269 >> v1:172.21.15.106:6789/0 conn(0x7f0c7c072430 legacy=0x7f0c7c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:07.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.888+0000 7f0c80c00640 1 -- 172.21.15.81:0/377798269 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c0c5270 con 0x7f0c7c161cd0 2024-05-27T07:00:07.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c80c00640 1 --2- 172.21.15.81:0/377798269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c161cd0 0x7f0c7c1640f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c06f550 tx=0x7f0c7401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bb0875c7673a44 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:07.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c7b400640 1 -- 172.21.15.81:0/377798269 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740a56c0 con 0x7f0c7c161cd0 2024-05-27T07:00:07.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c7b400640 1 -- 172.21.15.81:0/377798269 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c740a5860 con 0x7f0c7c161cd0 2024-05-27T07:00:07.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c7b400640 1 -- 172.21.15.81:0/377798269 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740a5c30 con 0x7f0c7c161cd0 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 -- 172.21.15.81:0/377798269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c161cd0 msgr2=0x7f0c7c1640f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 --2- 172.21.15.81:0/377798269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c161cd0 0x7f0c7c1640f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c06f550 tx=0x7f0c7401c9f0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 -- 172.21.15.81:0/377798269 shutdown_connections 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 --2- 172.21.15.81:0/377798269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c161cd0 0x7f0c7c1640f0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 --2- 172.21.15.81:0/377798269 >> v2:172.21.15.106:3300/0 conn(0x7f0c7c0c5b60 0x7f0c7c0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 -- 172.21.15.81:0/377798269 >> 172.21.15.81:0/377798269 conn(0x7f0c7c070700 msgr2=0x7f0c7c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 -- 172.21.15.81:0/377798269 shutdown_connections 2024-05-27T07:00:07.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.892+0000 7f0c82c00640 1 -- 172.21.15.81:0/377798269 wait complete. 2024-05-27T07:00:07.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 Processor -- start 2024-05-27T07:00:07.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- start start 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c7c0c5b60 0x7f0c7c145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 0x7f0c7c147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c0ce6f0 con 0x7f0c7c072430 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c0c7dd0 con 0x7f0c7c0c5b60 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c0c5610 con 0x7f0c7c161cd0 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43424/0 (socket says 172.21.15.81:43424) 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 -- 172.21.15.81:0/561051994 learned_addr learned my addr 172.21.15.81:0/561051994 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:07.889 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c73e00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c7c0c5b60 0x7f0c7c145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c80c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 0x7f0c7c147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 -- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 msgr2=0x7f0c7c147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 0x7f0c7c147fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 -- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c7c0c5b60 msgr2=0x7f0c7c145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c7c0c5b60 0x7f0c7c145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 -- 172.21.15.81:0/561051994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c1f5c70 con 0x7f0c7c072430 2024-05-27T07:00:07.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c80c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 0x7f0c7c147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c7be00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c3c0 tx=0x7f0c6c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c017070 con 0x7f0c7c072430 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0052d0 con 0x7f0c7c072430 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7c1f5ef0 con 0x7f0c7c072430 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c003840 con 0x7f0c7c072430 2024-05-27T07:00:07.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.896+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7c1f64b0 con 0x7f0c7c072430 2024-05-27T07:00:07.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7c06a560 con 0x7f0c7c072430 2024-05-27T07:00:07.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f0c6c01a070 con 0x7f0c7c072430 2024-05-27T07:00:07.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c78c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 0x7f0c58071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:07.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f0c6c08e7a0 con 0x7f0c7c072430 2024-05-27T07:00:07.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c73e00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 0x7f0c58071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:07.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.900+0000 7f0c73e00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 0x7f0c58071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c148cb0 tx=0x7f0c64008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:07.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:07.904+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f0c7c1f5c70 con 0x7f0c7c072430 2024-05-27T07:00:08.089 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.096+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c7c0c5940 con 0x7f0c7c072430 2024-05-27T07:00:08.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.096+0000 7f0c78c00640 1 -- 172.21.15.81:0/561051994 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f0c6c001090 con 0x7f0c7c072430 2024-05-27T07:00:08.090 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:08.090 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:08.091 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 msgr2=0x7f0c58071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 0x7f0c58071910 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c148cb0 tx=0x7f0c64008040 comp rx=0 tx=0).stop 2024-05-27T07:00:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 msgr2=0x7f0c7c1572b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c3c0 tx=0x7f0c6c00c880 comp rx=0 tx=0).stop 2024-05-27T07:00:08.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 shutdown_connections 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0c7c161cd0 0x7f0c7c147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f0c5806f400 0x7f0c58071910 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c7c0c5b60 0x7f0c7c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 --2- 172.21.15.81:0/561051994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c7c072430 0x7f0c7c1572b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 >> 172.21.15.81:0/561051994 conn(0x7f0c7c070700 msgr2=0x7f0c7c0c3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 shutdown_connections 2024-05-27T07:00:08.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:08.100+0000 7f0c82c00640 1 -- 172.21.15.81:0/561051994 wait complete. 2024-05-27T07:00:08.112 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:10.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:10.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 Processor -- start 2024-05-27T07:00:10.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 -- start start 2024-05-27T07:00:10.275 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4e4072430 0x7fe4e4159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e4072840 con 0x7fe4e415a040 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e40c5270 con 0x7fe4e4161cd0 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4eb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fe4e40c5040 con 0x7fe4e4072430 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.280+0000 7fe4e3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4e4072430 0x7fe4e4159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4db400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe4e415a040 0x7fe4e40c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37482/0 (socket says 172.21.15.81:37482) 2024-05-27T07:00:10.276 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4db400640 1 -- 172.21.15.81:0/253338433 learned_addr learned my addr 172.21.15.81:0/253338433 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4e4072430 0x7fe4e4159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43428/0 (socket says 172.21.15.81:43428) 2024-05-27T07:00:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3e00640 1 -- 172.21.15.81:0/253338433 >> v1:172.21.15.106:6789/0 conn(0x7fe4e4161cd0 legacy=0x7fe4e41640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3e00640 1 -- 172.21.15.81:0/253338433 >> v1:172.21.15.81:6789/0 conn(0x7fe4e415a040 legacy=0x7fe4e40c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3e00640 1 -- 172.21.15.81:0/253338433 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e40c7810 con 0x7fe4e4072430 2024-05-27T07:00:10.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3400640 1 -- 172.21.15.81:0/253338433 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710544468 0 0) 0x7fe4e4072840 con 0x7fe4e415a040 2024-05-27T07:00:10.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3e00640 1 --2- 172.21.15.81:0/253338433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e4159b00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40097c0 tx=0x7fe4d401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806cda6cd33ef83 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:10.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3400640 1 -- 172.21.15.81:0/253338433 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40b0070 con 0x7fe4e4072430 2024-05-27T07:00:10.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3400640 1 -- 172.21.15.81:0/253338433 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d40ab600 con 0x7fe4e4072430 2024-05-27T07:00:10.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.284+0000 7fe4e3400640 1 -- 172.21.15.81:0/253338433 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40ab9d0 con 0x7fe4e4072430 2024-05-27T07:00:10.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- 172.21.15.81:0/253338433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 msgr2=0x7fe4e4159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 --2- 172.21.15.81:0/253338433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e4159b00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40097c0 tx=0x7fe4d401c4a0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- 172.21.15.81:0/253338433 shutdown_connections 2024-05-27T07:00:10.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 --2- 172.21.15.81:0/253338433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e4159b00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- 172.21.15.81:0/253338433 >> 172.21.15.81:0/253338433 conn(0x7fe4e4070700 msgr2=0x7fe4e40cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:10.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- 172.21.15.81:0/253338433 shutdown_connections 2024-05-27T07:00:10.282 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- 172.21.15.81:0/253338433 wait complete. 2024-05-27T07:00:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 Processor -- start 2024-05-27T07:00:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.288+0000 7fe4eb200640 1 -- start start 2024-05-27T07:00:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4e415a040 0x7fe4e415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:10.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e4161cd0 0x7fe4e415c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:10.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e40c7dd0 con 0x7fe4e4072430 2024-05-27T07:00:10.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e40c75c0 con 0x7fe4e4161cd0 2024-05-27T07:00:10.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e40c5610 con 0x7fe4e415a040 2024-05-27T07:00:10.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:10.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43434/0 (socket says 172.21.15.81:43434) 2024-05-27T07:00:10.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 -- 172.21.15.81:0/3678371763 learned_addr learned my addr 172.21.15.81:0/3678371763 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:10.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 -- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4e415a040 msgr2=0x7fe4e415b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4e415a040 0x7fe4e415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 -- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e4161cd0 msgr2=0x7fe4e415c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e4161cd0 0x7fe4e415c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 -- 172.21.15.81:0/3678371763 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e41f7f60 con 0x7fe4e4072430 2024-05-27T07:00:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e3e00640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40ae040 tx=0x7fe4d401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:10.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40b0050 con 0x7fe4e4072430 2024-05-27T07:00:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d40bc070 con 0x7fe4e4072430 2024-05-27T07:00:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e41583e0 con 0x7fe4e4072430 2024-05-27T07:00:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40b5a70 con 0x7fe4e4072430 2024-05-27T07:00:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e41589a0 con 0x7fe4e4072430 2024-05-27T07:00:10.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e406a560 con 0x7fe4e4072430 2024-05-27T07:00:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.292+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fe4d40ab4c0 con 0x7fe4e4072430 2024-05-27T07:00:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.296+0000 7fe4e0c00640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 0x7fe4c0071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.296+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fe4d4129e20 con 0x7fe4e4072430 2024-05-27T07:00:10.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.296+0000 7fe4db400640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 0x7fe4c0071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.296+0000 7fe4db400640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 0x7fe4c0071940 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c10 tx=0x7fe4cc0079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:10.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.300+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fe4d4016090 con 0x7fe4e4072430 2024-05-27T07:00:10.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.492+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4e4050d90 con 0x7fe4e4072430 2024-05-27T07:00:10.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.492+0000 7fe4e0c00640 1 -- 172.21.15.81:0/3678371763 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fe4d40f6a80 con 0x7fe4e4072430 2024-05-27T07:00:10.487 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:10.487 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:10.487 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:10.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 msgr2=0x7fe4c0071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 0x7fe4c0071940 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc000c10 tx=0x7fe4cc0079d0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 msgr2=0x7fe4e41f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:10.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40ae040 tx=0x7fe4d401ace0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 shutdown_connections 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fe4e415a040 0x7fe4e415b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fe4c006f430 0x7fe4c0071940 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e4161cd0 0x7fe4e415c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 --2- 172.21.15.81:0/3678371763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e4072430 0x7fe4e41f7a20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:10.490 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 >> 172.21.15.81:0/3678371763 conn(0x7fe4e4070700 msgr2=0x7fe4e40c4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:10.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 shutdown_connections 2024-05-27T07:00:10.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:10.496+0000 7fe4eb200640 1 -- 172.21.15.81:0/3678371763 wait complete. 2024-05-27T07:00:10.509 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:12.510 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:12.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.676+0000 7f5d46000640 1 Processor -- start 2024-05-27T07:00:12.670 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.676+0000 7f5d46000640 1 -- start start 2024-05-27T07:00:12.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d46000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 0x7f5d40072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d46000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f5d400732f0 0x7f5d400c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d46000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4015db00 con 0x7f5d400c7510 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d46000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f5d4015e6f0 con 0x7f5d400732f0 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d46000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5d400755b0 con 0x7f5d401546f0 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3fe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5d400c7510 0x7f5d4015a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37498/0 (socket says 172.21.15.81:37498) 2024-05-27T07:00:12.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3fe00640 1 -- 172.21.15.81:0/1346200211 learned_addr learned my addr 172.21.15.81:0/1346200211 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3ea00640 1 --2- 172.21.15.81:0/1346200211 >> v2:172.21.15.106:3300/0 conn(0x7f5d400732f0 0x7f5d400c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3f400640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 0x7f5d40072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212668258 0 0) 0x7f5d4015db00 con 0x7f5d400c7510 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d24003530 con 0x7f5d400c7510 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3ea00640 1 -- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 msgr2=0x7f5d40072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4037295197 0 0) 0x7f5d24003530 con 0x7f5d400c7510 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3ea00640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 0x7f5d40072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3ea00640 1 -- 172.21.15.81:0/1346200211 >> v1:172.21.15.81:6789/0 conn(0x7f5d400c7510 legacy=0x7f5d4015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3ea00640 1 -- 172.21.15.81:0/1346200211 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d40073860 con 0x7f5d400732f0 2024-05-27T07:00:12.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.680+0000 7f5d3f400640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 0x7f5d40072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:12.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d3ea00640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400732f0 0x7f5d400c6fd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097c0 tx=0x7f5d2c01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59a00f675027ccf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c0b5070 con 0x7f5d400732f0 2024-05-27T07:00:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d2c0a85f0 con 0x7f5d400732f0 2024-05-27T07:00:12.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d3e000640 1 -- 172.21.15.81:0/1346200211 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38001700 con 0x7f5d400732f0 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 -- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400732f0 msgr2=0x7f5d400c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400732f0 0x7f5d400c6fd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097c0 tx=0x7f5d2c01ebb0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 -- 172.21.15.81:0/1346200211 shutdown_connections 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400732f0 0x7f5d400c6fd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 --2- 172.21.15.81:0/1346200211 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d401546f0 0x7f5d40072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 -- 172.21.15.81:0/1346200211 >> 172.21.15.81:0/1346200211 conn(0x7f5d40070700 msgr2=0x7f5d400cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 -- 172.21.15.81:0/1346200211 shutdown_connections 2024-05-27T07:00:12.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.684+0000 7f5d46000640 1 -- 172.21.15.81:0/1346200211 wait complete. 2024-05-27T07:00:12.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 Processor -- start 2024-05-27T07:00:12.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- start start 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d400732f0 0x7f5d400c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400c7510 0x7f5d400c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4015e380 con 0x7f5d401546f0 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4015e0a0 con 0x7f5d400c7510 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f5d40154b00 con 0x7f5d400732f0 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43436/0 (socket says 172.21.15.81:43436) 2024-05-27T07:00:12.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 -- 172.21.15.81:0/2130491789 learned_addr learned my addr 172.21.15.81:0/2130491789 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3f400640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d400732f0 0x7f5d400c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3ea00640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400c7510 0x7f5d400c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 -- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d400732f0 msgr2=0x7f5d400c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d400732f0 0x7f5d400c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 -- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400c7510 msgr2=0x7f5d400c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400c7510 0x7f5d400c9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 -- 172.21.15.81:0/2130491789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d400c6560 con 0x7f5d401546f0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d3fe00640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c500 tx=0x7f5d3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38005cf0 con 0x7f5d401546f0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d38007070 con 0x7f5d401546f0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38007460 con 0x7f5d401546f0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4014dd80 con 0x7f5d401546f0 2024-05-27T07:00:12.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.688+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d400c5230 con 0x7f5d401546f0 2024-05-27T07:00:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d4006a560 con 0x7f5d401546f0 2024-05-27T07:00:12.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f5d38007600 con 0x7f5d401546f0 2024-05-27T07:00:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d37400640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 0x7f5d20071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f5d3808f1c0 con 0x7f5d401546f0 2024-05-27T07:00:12.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d3f400640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 0x7f5d20071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:12.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.692+0000 7f5d3f400640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 0x7f5d20071940 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009bd0 tx=0x7f5d3000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:12.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.696+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f5d3805b7d0 con 0x7f5d401546f0 2024-05-27T07:00:12.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~248s 2024-05-27T07:00:12.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.888+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d40152df0 con 0x7f5d401546f0 2024-05-27T07:00:12.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.888+0000 7f5d37400640 1 -- 172.21.15.81:0/2130491789 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f5d3805fad0 con 0x7f5d401546f0 2024-05-27T07:00:12.882 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:12.882 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:12.882 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 msgr2=0x7f5d20071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 0x7f5d20071940 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5d30009bd0 tx=0x7f5d3000b040 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 msgr2=0x7f5d40155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c500 tx=0x7f5d3800c9c0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 shutdown_connections 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f5d2006f430 0x7f5d20071940 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d401546f0 0x7f5d40155f80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d400c7510 0x7f5d400c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 --2- 172.21.15.81:0/2130491789 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f5d400732f0 0x7f5d400c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 >> 172.21.15.81:0/2130491789 conn(0x7f5d40070700 msgr2=0x7f5d4015b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:12.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 shutdown_connections 2024-05-27T07:00:12.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:12.892+0000 7f5d46000640 1 -- 172.21.15.81:0/2130491789 wait complete. 2024-05-27T07:00:12.904 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:14.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:15.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.072+0000 7feea9600640 1 Processor -- start 2024-05-27T07:00:15.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.072+0000 7feea9600640 1 -- start start 2024-05-27T07:00:15.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea9600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7feea4073c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea9600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea406dd50 con 0x7feea40736c0 2024-05-27T07:00:15.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea9600640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea415d710 con 0x7feea406d120 2024-05-27T07:00:15.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea9600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7feea4074770 con 0x7feea4156c40 2024-05-27T07:00:15.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7fee9a000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7feea40736c0 0x7feea4074c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:37514/0 (socket says 172.21.15.81:37514) 2024-05-27T07:00:15.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7fee9a000640 1 -- 172.21.15.81:0/2190034534 learned_addr learned my addr 172.21.15.81:0/2190034534 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:15.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7feea4073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555466623 0 0) 0x7feea406dd50 con 0x7feea40736c0 2024-05-27T07:00:15.069 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee88003530 con 0x7feea40736c0 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1892521746 0 0) 0x7feea415d710 con 0x7feea406d120 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feea406dd50 con 0x7feea406d120 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3473593768 0 0) 0x7fee88003530 con 0x7feea40736c0 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 msgr2=0x7feea4073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7feea4073c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 >> v1:172.21.15.106:6789/0 conn(0x7feea406d120 legacy=0x7feea4167040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea406c8c0 con 0x7feea40736c0 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7fee8c001700 con 0x7feea40736c0 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 >> v1:172.21.15.81:6789/0 conn(0x7feea40736c0 legacy=0x7feea4074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7fee88004070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee88004810 0x7fee88004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7fee880052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feea4074770 con 0x7fee88003c20 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fee8800bbd0 con 0x7fee88004810 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fee8800bdd0 con 0x7feea4156c40 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7fee88004070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea3400640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7fee880052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7fee9a000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee88004810 0x7fee88004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 -- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 msgr2=0x7fee880052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7fee880052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 -- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee88004810 msgr2=0x7fee88004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee88004810 0x7fee88004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea2a00640 1 -- 172.21.15.81:0/2190034534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee88003530 con 0x7fee88003c20 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7feea3400640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea4156c40 0x7fee880052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.076+0000 7fee9a000640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee88004810 0x7fee88004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea2a00640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7fee88004070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fee94009ae0 tx=0x7fee9401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fec29a41395a6a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940096b0 con 0x7fee88003c20 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee940a6470 con 0x7fee88003c20 2024-05-27T07:00:15.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea2000640 1 -- 172.21.15.81:0/2190034534 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940a6860 con 0x7fee88003c20 2024-05-27T07:00:15.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 -- 172.21.15.81:0/2190034534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 msgr2=0x7fee88004070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 --2- 172.21.15.81:0/2190034534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7fee88004070 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fee94009ae0 tx=0x7fee9401ca90 comp rx=0 tx=0).stop 2024-05-27T07:00:15.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea3400640 1 -- 172.21.15.81:0/2190034534 reap_dead start 2024-05-27T07:00:15.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 -- 172.21.15.81:0/2190034534 shutdown_connections 2024-05-27T07:00:15.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 -- 172.21.15.81:0/2190034534 >> 172.21.15.81:0/2190034534 conn(0x7feea40cefd0 msgr2=0x7feea40cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:15.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 -- 172.21.15.81:0/2190034534 shutdown_connections 2024-05-27T07:00:15.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.080+0000 7feea9600640 1 -- 172.21.15.81:0/2190034534 wait complete. 2024-05-27T07:00:15.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 Processor -- start 2024-05-27T07:00:15.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- start start 2024-05-27T07:00:15.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 0x7feea40c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea41597f0 0x7feea40c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feea4075310 con 0x7fee88003c20 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feea406d940 con 0x7feea4159e10 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7feea4075490 con 0x7feea41597f0 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43450/0 (socket says 172.21.15.81:43450) 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 -- 172.21.15.81:0/317156539 learned_addr learned my addr 172.21.15.81:0/317156539 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7fee9a000640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 0x7feea40c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea3400640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea41597f0 0x7feea40c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 -- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea41597f0 msgr2=0x7feea40c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea41597f0 0x7feea40c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 -- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 msgr2=0x7feea40c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 0x7feea40c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 -- 172.21.15.81:0/317156539 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea4159c00 con 0x7fee88003c20 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7fee9a000640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 0x7feea40c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:15.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea2a00640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee94009ae0 tx=0x7fee94019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee940a6c20 con 0x7fee88003c20 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee940a6dc0 con 0x7fee88003c20 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea41608a0 con 0x7fee88003c20 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94004240 con 0x7fee88003c20 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.084+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea41f70e0 con 0x7fee88003c20 2024-05-27T07:00:15.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea406a560 con 0x7fee88003c20 2024-05-27T07:00:15.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fee94003620 con 0x7fee88003c20 2024-05-27T07:00:15.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7fee9b400640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 0x7fee80071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:15.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fee94129730 con 0x7fee88003c20 2024-05-27T07:00:15.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7fee9a000640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 0x7fee80071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:15.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.088+0000 7fee9a000640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 0x7fee80071940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee8c001ec0 tx=0x7fee8c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:15.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.092+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fee940c0060 con 0x7fee88003c20 2024-05-27T07:00:15.280 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.288+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea40c58d0 con 0x7fee88003c20 2024-05-27T07:00:15.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.288+0000 7fee9b400640 1 -- 172.21.15.81:0/317156539 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fee940b5070 con 0x7fee88003c20 2024-05-27T07:00:15.281 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:15.281 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:15.281 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:15.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 msgr2=0x7fee80071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 0x7fee80071940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fee8c001ec0 tx=0x7fee8c009040 comp rx=0 tx=0).stop 2024-05-27T07:00:15.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 msgr2=0x7feea415ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:15.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee94009ae0 tx=0x7fee94019830 comp rx=0 tx=0).stop 2024-05-27T07:00:15.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 shutdown_connections 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feea41597f0 0x7feea40c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fee8006f430 0x7fee80071940 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feea4159e10 0x7feea40c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 --2- 172.21.15.81:0/317156539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee88003c20 0x7feea415ddb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 >> 172.21.15.81:0/317156539 conn(0x7feea40cefd0 msgr2=0x7feea4154ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 shutdown_connections 2024-05-27T07:00:15.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:15.292+0000 7feea9600640 1 -- 172.21.15.81:0/317156539 wait complete. 2024-05-27T07:00:15.303 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:17.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:17.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 Processor -- start 2024-05-27T07:00:17.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 -- start start 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b801546f0 0x7f8b80072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f8b800732f0 0x7f8b800c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b800c7510 0x7f8b8015a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8015e6f0 con 0x7f8b801546f0 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f8b800755b0 con 0x7f8b800732f0 2024-05-27T07:00:17.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b86200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8015db00 con 0x7f8b800c7510 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b7f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b801546f0 0x7f8b80072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b7f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b801546f0 0x7f8b80072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43734/0 (socket says 172.21.15.81:43734) 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b7f400640 1 -- 172.21.15.81:0/265301536 learned_addr learned my addr 172.21.15.81:0/265301536 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b7fe00640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b800c7510 0x7f8b8015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.468+0000 7f8b76a00640 1 --2- 172.21.15.81:0/265301536 >> v2:172.21.15.106:3300/0 conn(0x7f8b800732f0 0x7f8b800c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 -- 172.21.15.81:0/265301536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b800c7510 msgr2=0x7f8b8015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b800c7510 0x7f8b8015a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 -- 172.21.15.81:0/265301536 >> v2:172.21.15.106:3300/0 conn(0x7f8b800732f0 msgr2=0x7f8b800c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 --2- 172.21.15.81:0/265301536 >> v2:172.21.15.106:3300/0 conn(0x7f8b800732f0 0x7f8b800c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 -- 172.21.15.81:0/265301536 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b80073860 con 0x7f8b801546f0 2024-05-27T07:00:17.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7f400640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b801546f0 0x7f8b80072db0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8b8006da10 tx=0x7f8b7001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c306ef62ec8f7cb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:17.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7ea00640 1 -- 172.21.15.81:0/265301536 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b700b4070 con 0x7f8b801546f0 2024-05-27T07:00:17.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7ea00640 1 -- 172.21.15.81:0/265301536 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b700af400 con 0x7f8b801546f0 2024-05-27T07:00:17.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.472+0000 7f8b7ea00640 1 -- 172.21.15.81:0/265301536 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b700af7d0 con 0x7f8b801546f0 2024-05-27T07:00:17.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- 172.21.15.81:0/265301536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b801546f0 msgr2=0x7f8b80072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:17.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b801546f0 0x7f8b80072db0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8b8006da10 tx=0x7f8b7001f1e0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- 172.21.15.81:0/265301536 shutdown_connections 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b800c7510 0x7f8b8015a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- 172.21.15.81:0/265301536 >> v2:172.21.15.106:3300/0 conn(0x7f8b800732f0 0x7f8b800c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- 172.21.15.81:0/265301536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b801546f0 0x7f8b80072db0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- 172.21.15.81:0/265301536 >> 172.21.15.81:0/265301536 conn(0x7f8b80070700 msgr2=0x7f8b800cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- 172.21.15.81:0/265301536 shutdown_connections 2024-05-27T07:00:17.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- 172.21.15.81:0/265301536 wait complete. 2024-05-27T07:00:17.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 Processor -- start 2024-05-27T07:00:17.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 -- start start 2024-05-27T07:00:17.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b800732f0 0x7f8b8015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.476+0000 7f8b86200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b801546f0 0x7f8b8014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8015e380 con 0x7f8b800c7510 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8015e0a0 con 0x7f8b800732f0 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f8b80154b00 con 0x7f8b801546f0 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43740/0 (socket says 172.21.15.81:43740) 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 -- 172.21.15.81:0/3790841263 learned_addr learned my addr 172.21.15.81:0/3790841263 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:17.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b7fe00640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b801546f0 0x7f8b8014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b7f400640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b800732f0 0x7f8b8015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 -- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b801546f0 msgr2=0x7f8b8014bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b801546f0 0x7f8b8014bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 -- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b800732f0 msgr2=0x7f8b8015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b800732f0 0x7f8b8015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 -- 172.21.15.81:0/3790841263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b801f3a50 con 0x7f8b800c7510 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b76a00640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800e8a0 tx=0x7f8b6800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:17.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68017070 con 0x7f8b800c7510 2024-05-27T07:00:17.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6800ee90 con 0x7f8b800c7510 2024-05-27T07:00:17.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800cb40 con 0x7f8b800c7510 2024-05-27T07:00:17.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8015a030 con 0x7f8b800c7510 2024-05-27T07:00:17.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b800cc990 con 0x7f8b800c7510 2024-05-27T07:00:17.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8006a560 con 0x7f8b800c7510 2024-05-27T07:00:17.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.480+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f8b6801a070 con 0x7f8b800c7510 2024-05-27T07:00:17.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.484+0000 7f8b77e00640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 0x7f8b5c071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:17.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.484+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f8b6808f320 con 0x7f8b800c7510 2024-05-27T07:00:17.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.484+0000 7f8b7f400640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 0x7f8b5c071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:17.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.484+0000 7f8b7f400640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 0x7f8b5c071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8b700b1000 tx=0x7f8b700aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.488+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f8b801f3a50 con 0x7f8b800c7510 2024-05-27T07:00:17.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.676+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b801f3d20 con 0x7f8b800c7510 2024-05-27T07:00:17.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.676+0000 7f8b77e00640 1 -- 172.21.15.81:0/3790841263 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f8b6805b930 con 0x7f8b800c7510 2024-05-27T07:00:17.670 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:17.670 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:17.670 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:17.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 msgr2=0x7f8b5c071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:17.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 0x7f8b5c071910 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8b700b1000 tx=0x7f8b700aa000 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 msgr2=0x7f8b800cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800e8a0 tx=0x7f8b6800ed60 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 shutdown_connections 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8b801546f0 0x7f8b8014bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f8b5c06f400 0x7f8b5c071910 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b800c7510 0x7f8b800cb930 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 --2- 172.21.15.81:0/3790841263 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b800732f0 0x7f8b8015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 >> 172.21.15.81:0/3790841263 conn(0x7f8b80070700 msgr2=0x7f8b800c5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 shutdown_connections 2024-05-27T07:00:17.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:17.680+0000 7f8b86200640 1 -- 172.21.15.81:0/3790841263 wait complete. 2024-05-27T07:00:17.692 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:00:19.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~255s 2024-05-27T07:00:19.693 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:19.854 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 Processor -- start 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 -- start start 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f46bc1546f0 0x7f46bc072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46bc15db00 con 0x7f46bc0732f0 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46bc073860 con 0x7f46bc15a510 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.864+0000 7f46c0c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f46bc15e6f0 con 0x7f46bc1546f0 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b1400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f46bc0732f0 0x7f46bc0c6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34192/0 (socket says 172.21.15.81:34192) 2024-05-27T07:00:19.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b1400640 1 -- 172.21.15.81:0/3490422525 learned_addr learned my addr 172.21.15.81:0/3490422525 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9e00640 1 --2- 172.21.15.81:0/3490422525 >> v2:172.21.15.106:3300/0 conn(0x7f46bc1546f0 0x7f46bc072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890147912 0 0) 0x7f46bc15db00 con 0x7f46bc0732f0 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46a0003530 con 0x7f46bc0732f0 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9e00640 1 -- 172.21.15.81:0/3490422525 >> v1:172.21.15.106:6789/0 conn(0x7f46bc15a510 legacy=0x7f46bc15a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4076463857 0 0) 0x7f46a0003530 con 0x7f46bc0732f0 2024-05-27T07:00:19.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9e00640 1 -- 172.21.15.81:0/3490422525 >> v1:172.21.15.81:6789/0 conn(0x7f46bc0732f0 legacy=0x7f46bc0c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:19.857 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.868+0000 7f46b9e00640 1 -- 172.21.15.81:0/3490422525 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc15ddc0 con 0x7f46bc1546f0 2024-05-27T07:00:19.858 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46b9e00640 1 --2- 172.21.15.81:0/3490422525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc1546f0 0x7f46bc072db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0097c0 tx=0x7f46ac01ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5955618f839c0436 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:19.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ac0b5070 con 0x7f46bc1546f0 2024-05-27T07:00:19.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ac0a8630 con 0x7f46bc1546f0 2024-05-27T07:00:19.859 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46b9400640 1 -- 172.21.15.81:0/3490422525 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4001700 con 0x7f46bc1546f0 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- 172.21.15.81:0/3490422525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc1546f0 msgr2=0x7f46bc072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3490422525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc1546f0 0x7f46bc072db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0097c0 tx=0x7f46ac01ebb0 comp rx=0 tx=0).stop 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- 172.21.15.81:0/3490422525 shutdown_connections 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3490422525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc1546f0 0x7f46bc072db0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- 172.21.15.81:0/3490422525 >> 172.21.15.81:0/3490422525 conn(0x7f46bc070700 msgr2=0x7f46bc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- 172.21.15.81:0/3490422525 shutdown_connections 2024-05-27T07:00:19.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- 172.21.15.81:0/3490422525 wait complete. 2024-05-27T07:00:19.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 Processor -- start 2024-05-27T07:00:19.862 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.872+0000 7f46c0c00640 1 -- start start 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc0732f0 0x7f46bc1f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46bc1546f0 0x7f46bc150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc15e380 con 0x7f46bc15a510 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f46bc0755b0 con 0x7f46bc0732f0 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f46bc154b00 con 0x7f46bc1546f0 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46b9e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc0732f0 0x7f46bc1f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43744/0 (socket says 172.21.15.81:43744) 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 -- 172.21.15.81:0/3386592736 learned_addr learned my addr 172.21.15.81:0/3386592736 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:19.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46b1400640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46bc1546f0 0x7f46bc150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 -- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46bc1546f0 msgr2=0x7f46bc150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46bc1546f0 0x7f46bc150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 -- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc0732f0 msgr2=0x7f46bc1f0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc0732f0 0x7f46bc1f0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 -- 172.21.15.81:0/3386592736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46bc1f1390 con 0x7f46bc15a510 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46ba800640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f46b400c6b0 tx=0x7f46b400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4003b80 con 0x7f46bc15a510 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b4003d20 con 0x7f46bc15a510 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46bc14e6c0 con 0x7f46bc15a510 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40052f0 con 0x7f46bc15a510 2024-05-27T07:00:19.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46bc1f43c0 con 0x7f46bc15a510 2024-05-27T07:00:19.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.876+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46bc14f210 con 0x7f46bc15a510 2024-05-27T07:00:19.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.880+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f46b4005490 con 0x7f46bc15a510 2024-05-27T07:00:19.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.880+0000 7f46b2a00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 0x7f4698071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:19.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.880+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f46b408f160 con 0x7f46bc15a510 2024-05-27T07:00:19.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.880+0000 7f46b9e00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 0x7f4698071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:19.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.880+0000 7f46b9e00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 0x7f4698071940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0b2000 tx=0x7f46ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:19.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:19.884+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f46b405b810 con 0x7f46bc15a510 2024-05-27T07:00:20.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46bc1f1560 con 0x7f46bc15a510 2024-05-27T07:00:20.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46b2a00640 1 -- 172.21.15.81:0/3386592736 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f46b405fb10 con 0x7f46bc15a510 2024-05-27T07:00:20.068 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:20.068 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:20.069 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:20.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 msgr2=0x7f4698071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:20.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 0x7f4698071940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46ac0b2000 tx=0x7f46ac008040 comp rx=0 tx=0).stop 2024-05-27T07:00:20.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 msgr2=0x7f46bc15c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:20.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f46b400c6b0 tx=0x7f46b400cb70 comp rx=0 tx=0).stop 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 shutdown_connections 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f46bc1546f0 0x7f46bc150ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f469806f430 0x7f4698071940 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46bc15a510 0x7f46bc15c0d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 --2- 172.21.15.81:0/3386592736 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f46bc0732f0 0x7f46bc1f0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 >> 172.21.15.81:0/3386592736 conn(0x7f46bc070700 msgr2=0x7f46bc14a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 shutdown_connections 2024-05-27T07:00:20.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:20.080+0000 7f46c0c00640 1 -- 172.21.15.81:0/3386592736 wait complete. 2024-05-27T07:00:20.090 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:22.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:22.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 Processor -- start 2024-05-27T07:00:22.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 -- start start 2024-05-27T07:00:22.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f14140ca380 0x7f14141566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14140ca950 con 0x7f141415fd90 2024-05-27T07:00:22.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f14140cb420 con 0x7f14140ca380 2024-05-27T07:00:22.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1418a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f14140cb670 con 0x7f14140cfd60 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1411e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f14140ca380 0x7f14141566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1409400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1411e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f14140ca380 0x7f14141566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:48592/0 (socket says 172.21.15.81:48592) 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1409400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:59742/0 (socket says 172.21.15.81:59742) 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.264+0000 7f1411e00640 1 -- 172.21.15.81:0/3285234666 learned_addr learned my addr 172.21.15.81:0/3285234666 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411e00640 1 -- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 msgr2=0x7f14140d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411e00640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411e00640 1 -- 172.21.15.81:0/3285234666 >> v1:172.21.15.106:6789/0 conn(0x7f141415fd90 legacy=0x7f14141601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411e00640 1 -- 172.21.15.81:0/3285234666 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14140cb950 con 0x7f14140ca380 2024-05-27T07:00:22.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411400640 1 -- 172.21.15.81:0/3285234666 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746606680 0 0) 0x7f14140ca950 con 0x7f141415fd90 2024-05-27T07:00:22.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1409400640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:22.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411e00640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140ca380 0x7f14141566e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f14040098a0 tx=0x7f140401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b977922a4c5ecb38 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:22.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411400640 1 -- 172.21.15.81:0/3285234666 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14040a9620 con 0x7f14140ca380 2024-05-27T07:00:22.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411400640 1 -- 172.21.15.81:0/3285234666 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14040a97c0 con 0x7f14140ca380 2024-05-27T07:00:22.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1411400640 1 -- 172.21.15.81:0/3285234666 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14040a9b90 con 0x7f14140ca380 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1418a00640 1 -- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140ca380 msgr2=0x7f14141566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.268+0000 7f1418a00640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140ca380 0x7f14141566e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f14040098a0 tx=0x7f140401c540 comp rx=0 tx=0).stop 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- 172.21.15.81:0/3285234666 shutdown_connections 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f14140cfd60 0x7f14140d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 --2- 172.21.15.81:0/3285234666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140ca380 0x7f14141566e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- 172.21.15.81:0/3285234666 >> 172.21.15.81:0/3285234666 conn(0x7f1414068ae0 msgr2=0x7f1414068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- 172.21.15.81:0/3285234666 shutdown_connections 2024-05-27T07:00:22.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- 172.21.15.81:0/3285234666 wait complete. 2024-05-27T07:00:22.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 Processor -- start 2024-05-27T07:00:22.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- start start 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140cfd60 0x7f141414b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f141415fd90 0x7f14140ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1414156da0 con 0x7f14140ca380 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1414156c20 con 0x7f14140cfd60 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1418a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1414169790 con 0x7f141415fd90 2024-05-27T07:00:22.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.272+0000 7f1411e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43746/0 (socket says 172.21.15.81:43746) 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 -- 172.21.15.81:0/2672285272 learned_addr learned my addr 172.21.15.81:0/2672285272 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1412800640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f141415fd90 0x7f14140ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1409400640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140cfd60 0x7f141414b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 -- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f141415fd90 msgr2=0x7f14140ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f141415fd90 0x7f14140ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 -- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140cfd60 msgr2=0x7f141414b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140cfd60 0x7f141414b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 -- 172.21.15.81:0/2672285272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141414bda0 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1411e00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f140401a170 tx=0x7f14040192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14040a9e40 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14040bb3f0 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14040bb7e0 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141415e360 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141415e920 con 0x7f14140ca380 2024-05-27T07:00:22.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141415eb60 con 0x7f14140ca380 2024-05-27T07:00:22.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.276+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f14040b13f0 con 0x7f14140ca380 2024-05-27T07:00:22.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.280+0000 7f140aa00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 0x7f13f0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:22.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.280+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f1404129390 con 0x7f14140ca380 2024-05-27T07:00:22.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.280+0000 7f1409400640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 0x7f13f0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:22.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.280+0000 7f1409400640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 0x7f13f0071910 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f141415bd20 tx=0x7f13fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:22.272 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.284+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f141415eb60 con 0x7f14140ca380 2024-05-27T07:00:22.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f141406a560 con 0x7f14140ca380 2024-05-27T07:00:22.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f140aa00640 1 -- 172.21.15.81:0/2672285272 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f14040b7070 con 0x7f14140ca380 2024-05-27T07:00:22.468 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:22.468 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:22.468 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 msgr2=0x7f13f0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 0x7f13f0071910 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f141415bd20 tx=0x7f13fc008040 comp rx=0 tx=0).stop 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 msgr2=0x7f141415f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f140401a170 tx=0x7f14040192e0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 shutdown_connections 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f141415fd90 0x7f14140ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f13f006f400 0x7f13f0071910 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14140cfd60 0x7f141414b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 --2- 172.21.15.81:0/2672285272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14140ca380 0x7f141415f470 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:22.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 >> 172.21.15.81:0/2672285272 conn(0x7f1414068ae0 msgr2=0x7f141415bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:22.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 shutdown_connections 2024-05-27T07:00:22.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:22.480+0000 7f1418a00640 1 -- 172.21.15.81:0/2672285272 wait complete. 2024-05-27T07:00:22.490 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:24.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 Processor -- start 2024-05-27T07:00:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 -- start start 2024-05-27T07:00:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f39040c6d40 0x7f39040c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39040c6a20 con 0x7f390415e6f0 2024-05-27T07:00:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390415db00 con 0x7f39040c7b20 2024-05-27T07:00:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390aa00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f39040c7f30 con 0x7f39040c6d40 2024-05-27T07:00:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390a000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f390415e6f0 0x7f3904164e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34206/0 (socket says 172.21.15.81:34206) 2024-05-27T07:00:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f390a000640 1 -- 172.21.15.81:0/649078352 learned_addr learned my addr 172.21.15.81:0/649078352 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3909600640 1 --2- 172.21.15.81:0/649078352 >> v2:172.21.15.106:3300/0 conn(0x7f39040c6d40 0x7f39040c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3903e00640 1 -- 172.21.15.81:0/649078352 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600360287 0 0) 0x7f39040c6a20 con 0x7f390415e6f0 2024-05-27T07:00:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3903e00640 1 -- 172.21.15.81:0/649078352 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ec003530 con 0x7f390415e6f0 2024-05-27T07:00:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3909600640 1 -- 172.21.15.81:0/649078352 >> v1:172.21.15.106:6789/0 conn(0x7f39040c7b20 legacy=0x7f390415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3909600640 1 -- 172.21.15.81:0/649078352 >> v1:172.21.15.81:6789/0 conn(0x7f390415e6f0 legacy=0x7f3904164e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.664+0000 7f3909600640 1 -- 172.21.15.81:0/649078352 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39040c9d40 con 0x7f39040c6d40 2024-05-27T07:00:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f3909600640 1 --2- 172.21.15.81:0/649078352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c6d40 0x7f39040c75e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f38f80097c0 tx=0x7f38f801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13e9cafa647331fd server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f3903e00640 1 -- 172.21.15.81:0/649078352 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b5070 con 0x7f39040c6d40 2024-05-27T07:00:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f3903e00640 1 -- 172.21.15.81:0/649078352 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f80a8990 con 0x7f39040c6d40 2024-05-27T07:00:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f3903e00640 1 -- 172.21.15.81:0/649078352 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f80a8d60 con 0x7f39040c6d40 2024-05-27T07:00:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 -- 172.21.15.81:0/649078352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c6d40 msgr2=0x7f39040c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 --2- 172.21.15.81:0/649078352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c6d40 0x7f39040c75e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f38f80097c0 tx=0x7f38f801ec90 comp rx=0 tx=0).stop 2024-05-27T07:00:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 -- 172.21.15.81:0/649078352 shutdown_connections 2024-05-27T07:00:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 --2- 172.21.15.81:0/649078352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c6d40 0x7f39040c75e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 -- 172.21.15.81:0/649078352 >> 172.21.15.81:0/649078352 conn(0x7f3904074550 msgr2=0x7f3904074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 -- 172.21.15.81:0/649078352 shutdown_connections 2024-05-27T07:00:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.668+0000 7f390aa00640 1 -- 172.21.15.81:0/649078352 wait complete. 2024-05-27T07:00:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 Processor -- start 2024-05-27T07:00:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 -- start start 2024-05-27T07:00:24.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 0x7f3904153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f390415e6f0 0x7f39041e56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39040ca360 con 0x7f39040c6d40 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f39040c9a60 con 0x7f39040c7b20 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f390aa00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f39040ca640 con 0x7f390415e6f0 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f3909600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f3909600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43758/0 (socket says 172.21.15.81:43758) 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f3909600640 1 -- 172.21.15.81:0/2155094195 learned_addr learned my addr 172.21.15.81:0/2155094195 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.672+0000 7f3908c00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 0x7f3904153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 -- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f390415e6f0 msgr2=0x7f39041e56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f390415e6f0 0x7f39041e56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 -- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 msgr2=0x7f3904153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 0x7f3904153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 -- 172.21.15.81:0/2155094195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3904152a90 con 0x7f39040c6d40 2024-05-27T07:00:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3908c00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 0x7f3904153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3909600640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f38f801b8a0 tx=0x7f38f801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f80bf070 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f80b0720 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3904153590 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b5040 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39041f0e30 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3904050870 con 0x7f39040c6d40 2024-05-27T07:00:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f38f80c2070 con 0x7f39040c6d40 2024-05-27T07:00:24.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.676+0000 7f3901600640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 0x7f38e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:24.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.680+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f38f812bcd0 con 0x7f39040c6d40 2024-05-27T07:00:24.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.680+0000 7f3908c00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 0x7f38e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:24.667 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.680+0000 7f3908c00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 0x7f38e4071940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f39040cbfe0 tx=0x7f38f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:24.669 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.680+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f38f80187f0 con 0x7f39040c6d40 2024-05-27T07:00:24.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39040c6540 con 0x7f39040c6d40 2024-05-27T07:00:24.860 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f3901600640 1 -- 172.21.15.81:0/2155094195 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f38f80f89b0 con 0x7f39040c6d40 2024-05-27T07:00:24.861 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:24.861 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:24.861 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 msgr2=0x7f38e4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 0x7f38e4071940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f39040cbfe0 tx=0x7f38f0009040 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 msgr2=0x7f3904152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f38f801b8a0 tx=0x7f38f801ba30 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 shutdown_connections 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f390415e6f0 0x7f39041e56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.872+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f38e406f430 0x7f38e4071940 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.876+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39040c7b20 0x7f3904153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.876+0000 7f390aa00640 1 --2- 172.21.15.81:0/2155094195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39040c6d40 0x7f3904152550 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.876+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 >> 172.21.15.81:0/2155094195 conn(0x7f3904074550 msgr2=0x7f3904072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.876+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 shutdown_connections 2024-05-27T07:00:24.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:24.876+0000 7f390aa00640 1 -- 172.21.15.81:0/2155094195 wait complete. 2024-05-27T07:00:24.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:26.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~262s 2024-05-27T07:00:26.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:27.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.052+0000 7f30f4a00640 1 Processor -- start 2024-05-27T07:00:27.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.052+0000 7f30f4a00640 1 -- start start 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30f4a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30f4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f00c9f80 con 0x7f30f015cb80 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30f4a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f00c52f0 con 0x7f30f00c8880 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30f4a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f30f00c58b0 con 0x7f30f006c790 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ede00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f30f015cb80 0x7f30f0152cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60880/0 (socket says 172.21.15.81:60880) 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ede00640 1 -- 172.21.15.81:0/1816236917 learned_addr learned my addr 172.21.15.81:0/1816236917 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30e5400640 1 --2- 172.21.15.81:0/1816236917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648577871 0 0) 0x7f30f00c9f80 con 0x7f30f015cb80 2024-05-27T07:00:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30d4004640 con 0x7f30f015cb80 2024-05-27T07:00:27.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2986456149 0 0) 0x7f30f00c52f0 con 0x7f30f00c8880 2024-05-27T07:00:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30f00c9f80 con 0x7f30f00c8880 2024-05-27T07:00:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30e5400640 1 -- 172.21.15.81:0/1816236917 >> v1:172.21.15.106:6789/0 conn(0x7f30f00c8880 legacy=0x7f30f00c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30e5400640 1 -- 172.21.15.81:0/1816236917 >> v1:172.21.15.81:6789/0 conn(0x7f30f015cb80 legacy=0x7f30f0152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.056+0000 7f30e5400640 1 -- 172.21.15.81:0/1816236917 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f00c55d0 con 0x7f30f006c790 2024-05-27T07:00:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30e5400640 1 --2- 172.21.15.81:0/1816236917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f006cba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f30d80097f0 tx=0x7f30d801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3454318435eba41 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80b0070 con 0x7f30f006c790 2024-05-27T07:00:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d80ab600 con 0x7f30f006c790 2024-05-27T07:00:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30ed400640 1 -- 172.21.15.81:0/1816236917 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80ab9d0 con 0x7f30f006c790 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 -- 172.21.15.81:0/1816236917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 msgr2=0x7f30f006cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 --2- 172.21.15.81:0/1816236917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f006cba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f30d80097f0 tx=0x7f30d801c4e0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 -- 172.21.15.81:0/1816236917 shutdown_connections 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 --2- 172.21.15.81:0/1816236917 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f006cba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 -- 172.21.15.81:0/1816236917 >> 172.21.15.81:0/1816236917 conn(0x7f30f0070700 msgr2=0x7f30f00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 -- 172.21.15.81:0/1816236917 shutdown_connections 2024-05-27T07:00:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.060+0000 7f30f4a00640 1 -- 172.21.15.81:0/1816236917 wait complete. 2024-05-27T07:00:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 Processor -- start 2024-05-27T07:00:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- start start 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f01f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 0x7f30f00c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30f00c9d10 con 0x7f30f00c8880 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f30f00c83f0 con 0x7f30f015cb80 2024-05-27T07:00:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f30f0166710 con 0x7f30f006c790 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55218/0 (socket says 172.21.15.81:55218) 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 -- 172.21.15.81:0/2783666241 learned_addr learned my addr 172.21.15.81:0/2783666241 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30ede00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f01f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30ee800640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 0x7f30f00c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 -- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 msgr2=0x7f30f01f5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f01f5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 -- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 msgr2=0x7f30f00c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 0x7f30f00c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 -- 172.21.15.81:0/2783666241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f01f6510 con 0x7f30f00c8880 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30ede00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f01f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30ee800640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 0x7f30f00c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e5400640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30d80ae040 tx=0x7f30d8019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80bb070 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d8004030 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f01545d0 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d80b0040 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.064+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f0154010 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f006a560 con 0x7f30f00c8880 2024-05-27T07:00:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f30d80be070 con 0x7f30f00c8880 2024-05-27T07:00:27.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30e6a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 0x7f30cc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:27.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f30d81284d0 con 0x7f30f00c8880 2024-05-27T07:00:27.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30ede00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 0x7f30cc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:27.058 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.068+0000 7f30ede00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 0x7f30cc071910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f30e00017b0 tx=0x7f30e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:27.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.076+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f30d80160d0 con 0x7f30f00c8880 2024-05-27T07:00:27.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.264+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30f00740a0 con 0x7f30f00c8880 2024-05-27T07:00:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30e6a00640 1 -- 172.21.15.81:0/2783666241 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f30d80f5120 con 0x7f30f00c8880 2024-05-27T07:00:27.255 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:27.255 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 msgr2=0x7f30cc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 0x7f30cc071910 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f30e00017b0 tx=0x7f30e0009040 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 msgr2=0x7f30f014e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f30d80ae040 tx=0x7f30d8019280 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 shutdown_connections 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f30cc06f400 0x7f30cc071910 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30f015cb80 0x7f30f00c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f00c8880 0x7f30f014e950 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 --2- 172.21.15.81:0/2783666241 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f30f006c790 0x7f30f01f5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 >> 172.21.15.81:0/2783666241 conn(0x7f30f0070700 msgr2=0x7f30f00cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 shutdown_connections 2024-05-27T07:00:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:27.268+0000 7f30f4a00640 1 -- 172.21.15.81:0/2783666241 wait complete. 2024-05-27T07:00:27.278 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:29.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:29.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 Processor -- start 2024-05-27T07:00:29.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 -- start start 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d2c0c96b0 0x7f3d2c156820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 0x7f3d2c160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2c15cb60 con 0x7f3d2c0cebf0 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2c15ce40 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d31e00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2c15d120 con 0x7f3d2c160370 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d2c0c96b0 0x7f3d2c156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d2c0c96b0 0x7f3d2c156820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55228/0 (socket says 172.21.15.81:55228) 2024-05-27T07:00:29.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 -- 172.21.15.81:0/2723174725 learned_addr learned my addr 172.21.15.81:0/2723174725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:29.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 -- 172.21.15.81:0/2723174725 >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 msgr2=0x7f3d2c160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:29.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d2be00640 1 --2- 172.21.15.81:0/2723174725 >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 0x7f3d2c160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 --2- 172.21.15.81:0/2723174725 >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 0x7f3d2c160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 -- 172.21.15.81:0/2723174725 >> v1:172.21.15.106:6789/0 conn(0x7f3d2c0cebf0 legacy=0x7f3d2c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:29.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d30a00640 1 -- 172.21.15.81:0/2723174725 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c15d400 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.452+0000 7f3d2be00640 1 --2- 172.21.15.81:0/2723174725 >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 0x7f3d2c160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:29.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d30a00640 1 --2- 172.21.15.81:0/2723174725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c156820 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c073350 tx=0x7f3d1c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b7f94757c0083c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:29.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d2b400640 1 -- 172.21.15.81:0/2723174725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0b4070 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d2b400640 1 -- 172.21.15.81:0/2723174725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0af400 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d2b400640 1 -- 172.21.15.81:0/2723174725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0af7d0 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 -- 172.21.15.81:0/2723174725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c0c96b0 msgr2=0x7f3d2c156820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 --2- 172.21.15.81:0/2723174725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c156820 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c073350 tx=0x7f3d1c01f1e0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 -- 172.21.15.81:0/2723174725 shutdown_connections 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 --2- 172.21.15.81:0/2723174725 >> v2:172.21.15.106:3300/0 conn(0x7f3d2c160370 0x7f3d2c160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 --2- 172.21.15.81:0/2723174725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c156820 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 -- 172.21.15.81:0/2723174725 >> 172.21.15.81:0/2723174725 conn(0x7f3d2c071390 msgr2=0x7f3d2c0717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 -- 172.21.15.81:0/2723174725 shutdown_connections 2024-05-27T07:00:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.456+0000 7f3d31e00640 1 -- 172.21.15.81:0/2723174725 wait complete. 2024-05-27T07:00:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 Processor -- start 2024-05-27T07:00:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 -- start start 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c1f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 0x7f3d2c0c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c156ee0 con 0x7f3d2c160370 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c156d60 con 0x7f3d2c0c96b0 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c15c720 con 0x7f3d2c0cebf0 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55234/0 (socket says 172.21.15.81:55234) 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 -- 172.21.15.81:0/210076900 learned_addr learned my addr 172.21.15.81:0/210076900 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d30a00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c1f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d2be00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 0x7f3d2c0c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 -- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 msgr2=0x7f3d2c0c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 0x7f3d2c0c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 -- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d2c0c96b0 msgr2=0x7f3d2c1f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c1f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 -- 172.21.15.81:0/210076900 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c1e9400 con 0x7f3d2c160370 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d2be00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 0x7f3d2c0c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.460+0000 7f3d31400640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b470 tx=0x7f3d2400b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24015020 con 0x7f3d2c160370 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d24011890 con 0x7f3d2c160370 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24011c80 con 0x7f3d2c160370 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c0c7f00 con 0x7f3d2c160370 2024-05-27T07:00:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c0c84c0 con 0x7f3d2c160370 2024-05-27T07:00:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c0c7c70 con 0x7f3d2c160370 2024-05-27T07:00:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f3d24018020 con 0x7f3d2c160370 2024-05-27T07:00:29.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 0x7f3d0c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:29.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f3d2408f250 con 0x7f3d2c160370 2024-05-27T07:00:29.454 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.464+0000 7f3d30a00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 0x7f3d0c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:29.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.468+0000 7f3d30a00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 0x7f3d0c071940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b1000 tx=0x7f3d1c0aa000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:29.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.472+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f3d2401f030 con 0x7f3d2c160370 2024-05-27T07:00:29.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d2c1e8820 con 0x7f3d2c160370 2024-05-27T07:00:29.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d28c00640 1 -- 172.21.15.81:0/210076900 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f3d2400d040 con 0x7f3d2c160370 2024-05-27T07:00:29.652 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:29.652 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:29.652 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 msgr2=0x7f3d0c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 0x7f3d0c071940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0b1000 tx=0x7f3d1c0aa000 comp rx=0 tx=0).stop 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 msgr2=0x7f3d2c1e8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b470 tx=0x7f3d2400b930 comp rx=0 tx=0).stop 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 shutdown_connections 2024-05-27T07:00:29.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f3d2c0cebf0 0x7f3d2c0c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f3d0c06f430 0x7f3d0c071940 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d2c160370 0x7f3d2c1e8ec0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 --2- 172.21.15.81:0/210076900 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d2c0c96b0 0x7f3d2c1f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 >> 172.21.15.81:0/210076900 conn(0x7f3d2c071390 msgr2=0x7f3d2c15bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 shutdown_connections 2024-05-27T07:00:29.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:29.664+0000 7f3d31e00640 1 -- 172.21.15.81:0/210076900 wait complete. 2024-05-27T07:00:29.675 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:00:31.677 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:31.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.848+0000 7f1b5d600640 1 Processor -- start 2024-05-27T07:00:31.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.848+0000 7f1b5d600640 1 -- start start 2024-05-27T07:00:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 0x7f1b58072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1b580732f0 0x7f1b580c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b5815a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f1b5815e6f0 con 0x7f1b580732f0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1b580755b0 con 0x7f1b581546f0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b5d600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5815db00 con 0x7f1b580c7510 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1b580732f0 0x7f1b580c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1b580732f0 0x7f1b580c6fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55248/0 (socket says 172.21.15.81:55248) 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 -- 172.21.15.81:0/3977810355 learned_addr learned my addr 172.21.15.81:0/3977810355 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 -- 172.21.15.81:0/3977810355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 msgr2=0x7f1b5815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b57400640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b5815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b56a00640 1 --2- 172.21.15.81:0/3977810355 >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 0x7f1b58072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b5815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 -- 172.21.15.81:0/3977810355 >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 msgr2=0x7f1b58072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 --2- 172.21.15.81:0/3977810355 >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 0x7f1b58072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 -- 172.21.15.81:0/3977810355 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b58073860 con 0x7f1b580732f0 2024-05-27T07:00:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b56a00640 1 --2- 172.21.15.81:0/3977810355 >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 0x7f1b58072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:31.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b57400640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b5815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:31.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.852+0000 7f1b4e000640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b580732f0 0x7f1b580c6fd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806d950 tx=0x7f1b4001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c68801d6fff5b server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:31.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b56000640 1 -- 172.21.15.81:0/3977810355 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b400b5070 con 0x7f1b580732f0 2024-05-27T07:00:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b56000640 1 -- 172.21.15.81:0/3977810355 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b400a89d0 con 0x7f1b580732f0 2024-05-27T07:00:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b56000640 1 -- 172.21.15.81:0/3977810355 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b400a8da0 con 0x7f1b580732f0 2024-05-27T07:00:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- 172.21.15.81:0/3977810355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b580732f0 msgr2=0x7f1b580c6fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b580732f0 0x7f1b580c6fd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f1b5806d950 tx=0x7f1b4001ed10 comp rx=0 tx=0).stop 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- 172.21.15.81:0/3977810355 shutdown_connections 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b5815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 --2- 172.21.15.81:0/3977810355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b580732f0 0x7f1b580c6fd0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 --2- 172.21.15.81:0/3977810355 >> v2:172.21.15.106:3300/0 conn(0x7f1b581546f0 0x7f1b58072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- 172.21.15.81:0/3977810355 >> 172.21.15.81:0/3977810355 conn(0x7f1b58070700 msgr2=0x7f1b580cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- 172.21.15.81:0/3977810355 shutdown_connections 2024-05-27T07:00:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- 172.21.15.81:0/3977810355 wait complete. 2024-05-27T07:00:31.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 Processor -- start 2024-05-27T07:00:31.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.856+0000 7f1b5d600640 1 -- start start 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b580732f0 0x7f1b5814ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b580c3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5815e380 con 0x7f1b581546f0 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5815e0a0 con 0x7f1b580732f0 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1b58154b00 con 0x7f1b580c7510 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55252/0 (socket says 172.21.15.81:55252) 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 -- 172.21.15.81:0/4027974794 learned_addr learned my addr 172.21.15.81:0/4027974794 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b4e000640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b580c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b56a00640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b580732f0 0x7f1b5814ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 -- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 msgr2=0x7f1b580c3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b580c3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 -- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b580732f0 msgr2=0x7f1b5814ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b580732f0 0x7f1b5814ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 -- 172.21.15.81:0/4027974794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b580cdea0 con 0x7f1b581546f0 2024-05-27T07:00:31.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b57400640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000cd80 tx=0x7f1b5000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000f680 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b5000f840 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000fc30 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b581e5ba0 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b581e5270 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.860+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5806a560 con 0x7f1b581546f0 2024-05-27T07:00:31.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.864+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1b5000e3e0 con 0x7f1b581546f0 2024-05-27T07:00:31.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.864+0000 7f1b4f400640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 0x7f1b34071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.864+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f1b50090460 con 0x7f1b581546f0 2024-05-27T07:00:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.864+0000 7f1b56a00640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 0x7f1b34071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:31.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.864+0000 7f1b56a00640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 0x7f1b34071910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006fd0 tx=0x7f1b48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:31.855 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:31.868+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1b50014030 con 0x7f1b581546f0 2024-05-27T07:00:32.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.056+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b580c8e20 con 0x7f1b581546f0 2024-05-27T07:00:32.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b4f400640 1 -- 172.21.15.81:0/4027974794 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f1b5005d140 con 0x7f1b581546f0 2024-05-27T07:00:32.047 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:32.047 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:32.047 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:32.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 msgr2=0x7f1b34071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:32.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 0x7f1b34071910 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006fd0 tx=0x7f1b48008040 comp rx=0 tx=0).stop 2024-05-27T07:00:32.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 msgr2=0x7f1b580c3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:32.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000cd80 tx=0x7f1b5000a650 comp rx=0 tx=0).stop 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 shutdown_connections 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1b580c7510 0x7f1b580c3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1b3406f400 0x7f1b34071910 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b581546f0 0x7f1b580c3a60 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 --2- 172.21.15.81:0/4027974794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1b580732f0 0x7f1b5814ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 >> 172.21.15.81:0/4027974794 conn(0x7f1b58070700 msgr2=0x7f1b58151400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 shutdown_connections 2024-05-27T07:00:32.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:32.060+0000 7f1b5d600640 1 -- 172.21.15.81:0/4027974794 wait complete. 2024-05-27T07:00:32.068 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:32.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~269s 2024-05-27T07:00:34.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:34.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.240+0000 7fd5df200640 1 Processor -- start 2024-05-27T07:00:34.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.240+0000 7fd5df200640 1 -- start start 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5df200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 0x7fd5d8154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5df200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5d815ded0 0x7fd5d8164e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5df200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5d80c9c70 con 0x7fd5d815dac0 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5df200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd5d80c9a40 con 0x7fd5d815ded0 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5df200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd5d815b0c0 con 0x7fd5d80c9630 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5de800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5d815ded0 0x7fd5d8164e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5dd400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd5d815dac0 0x7fd5d8072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:60896/0 (socket says 172.21.15.81:60896) 2024-05-27T07:00:34.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5dd400640 1 -- 172.21.15.81:0/3081466192 learned_addr learned my addr 172.21.15.81:0/3081466192 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3081466192 >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 0x7fd5d8154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5de800640 1 -- 172.21.15.81:0/3081466192 >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 msgr2=0x7fd5d8154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5de800640 1 --2- 172.21.15.81:0/3081466192 >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 0x7fd5d8154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5de800640 1 -- 172.21.15.81:0/3081466192 >> v1:172.21.15.81:6789/0 conn(0x7fd5d815dac0 legacy=0x7fd5d8072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5de800640 1 -- 172.21.15.81:0/3081466192 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d815a4d0 con 0x7fd5d815ded0 2024-05-27T07:00:34.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5dca00640 1 -- 172.21.15.81:0/3081466192 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864102100 0 0) 0x7fd5d80c9c70 con 0x7fd5d815dac0 2024-05-27T07:00:34.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.244+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3081466192 >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 0x7fd5d8154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:34.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5de800640 1 --2- 172.21.15.81:0/3081466192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815ded0 0x7fd5d8164e50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cc4a0 tx=0x7fd5d401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6168084fe454c1a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:34.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5dca00640 1 -- 172.21.15.81:0/3081466192 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40b2070 con 0x7fd5d815ded0 2024-05-27T07:00:34.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5dca00640 1 -- 172.21.15.81:0/3081466192 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d40a5960 con 0x7fd5d815ded0 2024-05-27T07:00:34.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5dca00640 1 -- 172.21.15.81:0/3081466192 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d40a5d30 con 0x7fd5d815ded0 2024-05-27T07:00:34.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 -- 172.21.15.81:0/3081466192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815ded0 msgr2=0x7fd5d8164e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 --2- 172.21.15.81:0/3081466192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815ded0 0x7fd5d8164e50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80cc4a0 tx=0x7fd5d401c8a0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 -- 172.21.15.81:0/3081466192 shutdown_connections 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 --2- 172.21.15.81:0/3081466192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815ded0 0x7fd5d8164e50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 --2- 172.21.15.81:0/3081466192 >> v2:172.21.15.106:3300/0 conn(0x7fd5d80c9630 0x7fd5d8154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 -- 172.21.15.81:0/3081466192 >> 172.21.15.81:0/3081466192 conn(0x7fd5d8068880 msgr2=0x7fd5d8068c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 -- 172.21.15.81:0/3081466192 shutdown_connections 2024-05-27T07:00:34.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.248+0000 7fd5df200640 1 -- 172.21.15.81:0/3081466192 wait complete. 2024-05-27T07:00:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 Processor -- start 2024-05-27T07:00:34.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- start start 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 0x7fd5d81f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5d815ded0 0x7fd5d814ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d815e2e0 con 0x7fd5d815dac0 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d80cd280 con 0x7fd5d815ded0 2024-05-27T07:00:34.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d8072c30 con 0x7fd5d80c9630 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55282/0 (socket says 172.21.15.81:55282) 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 -- 172.21.15.81:0/3634691912 learned_addr learned my addr 172.21.15.81:0/3634691912 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5de800640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5d815ded0 0x7fd5d814ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 0x7fd5d81f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 -- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 msgr2=0x7fd5d81f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 0x7fd5d81f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 -- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5d815ded0 msgr2=0x7fd5d814ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5d815ded0 0x7fd5d814ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 -- 172.21.15.81:0/3634691912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d81f6b00 con 0x7fd5d815dac0 2024-05-27T07:00:34.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 0x7fd5d81f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5dd400640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c440 tx=0x7fd5c800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8017070 con 0x7fd5d815dac0 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5c8005cf0 con 0x7fd5d815dac0 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c80072f0 con 0x7fd5d815dac0 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d81f9ab0 con 0x7fd5d815dac0 2024-05-27T07:00:34.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.252+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d81fa070 con 0x7fd5d815dac0 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d8050870 con 0x7fd5d815dac0 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd5c801a070 con 0x7fd5d815dac0 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5d2000640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 0x7fd5bc071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd5c808e750 con 0x7fd5d815dac0 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 0x7fd5bc071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:34.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.256+0000 7fd5dde00640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 0x7fd5bc071940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80c6cb0 tx=0x7fd5cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:34.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.260+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd5c805adf0 con 0x7fd5d815dac0 2024-05-27T07:00:34.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.464+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5d81f9840 con 0x7fd5d815dac0 2024-05-27T07:00:34.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.464+0000 7fd5d2000640 1 -- 172.21.15.81:0/3634691912 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd5c805f0f0 con 0x7fd5d815dac0 2024-05-27T07:00:34.453 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:34.453 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:34.454 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 msgr2=0x7fd5bc071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 0x7fd5bc071940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd5d80c6cb0 tx=0x7fd5cc008040 comp rx=0 tx=0).stop 2024-05-27T07:00:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 msgr2=0x7fd5d80cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800c440 tx=0x7fd5c800c900 comp rx=0 tx=0).stop 2024-05-27T07:00:34.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 shutdown_connections 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd5bc06f430 0x7fd5bc071940 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5d815ded0 0x7fd5d814ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5d815dac0 0x7fd5d80cea30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 --2- 172.21.15.81:0/3634691912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd5d80c9630 0x7fd5d81f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 >> 172.21.15.81:0/3634691912 conn(0x7fd5d8068880 msgr2=0x7fd5d81502a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 shutdown_connections 2024-05-27T07:00:34.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:34.468+0000 7fd5df200640 1 -- 172.21.15.81:0/3634691912 wait complete. 2024-05-27T07:00:34.475 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:35.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:00:35.175 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:00:36.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.648+0000 7ff696800640 1 Processor -- start 2024-05-27T07:00:36.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.648+0000 7ff696800640 1 -- start start 2024-05-27T07:00:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff696800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff690072430 0x7ff690159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff696800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff696800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff690072840 con 0x7ff6900c60a0 2024-05-27T07:00:36.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff696800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7ff6900c5040 con 0x7ff690072430 2024-05-27T07:00:36.639 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff696800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6900c75c0 con 0x7ff69015a040 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff686a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68f400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7ff690072430 0x7ff690159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff686a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:34804/0 (socket says 172.21.15.81:34804) 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff686a00640 1 -- 172.21.15.81:0/3356422183 learned_addr learned my addr 172.21.15.81:0/3356422183 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68fe00640 1 --1- >> v1:172.21.15.106:6789/0 conn(0x7ff6900c60a0 0x7ff69015c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:37960/0 (socket says 172.21.15.81:37960) 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68f400640 1 -- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 msgr2=0x7ff6900c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68f400640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68f400640 1 -- 172.21.15.81:0/3356422183 >> v1:172.21.15.106:6789/0 conn(0x7ff6900c60a0 legacy=0x7ff69015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68f400640 1 -- 172.21.15.81:0/3356422183 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6900c5270 con 0x7ff690072430 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff68ea00640 1 -- 172.21.15.81:0/3356422183 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813043020 0 0) 0x7ff690072840 con 0x7ff6900c60a0 2024-05-27T07:00:36.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.652+0000 7ff686a00640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:36.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff68f400640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff690072430 0x7ff690159b00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6800097c0 tx=0x7ff68001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8591902e18dd8bd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:36.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff68ea00640 1 -- 172.21.15.81:0/3356422183 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800b0070 con 0x7ff690072430 2024-05-27T07:00:36.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff68ea00640 1 -- 172.21.15.81:0/3356422183 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6800a4e50 con 0x7ff690072430 2024-05-27T07:00:36.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff68ea00640 1 -- 172.21.15.81:0/3356422183 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800ab820 con 0x7ff690072430 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 -- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff690072430 msgr2=0x7ff690159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff690072430 0x7ff690159b00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6800097c0 tx=0x7ff68001c4a0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 -- 172.21.15.81:0/3356422183 shutdown_connections 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6900c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 --2- 172.21.15.81:0/3356422183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff690072430 0x7ff690159b00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 -- 172.21.15.81:0/3356422183 >> 172.21.15.81:0/3356422183 conn(0x7ff690070700 msgr2=0x7ff6900cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:36.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 -- 172.21.15.81:0/3356422183 shutdown_connections 2024-05-27T07:00:36.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.656+0000 7ff696800640 1 -- 172.21.15.81:0/3356422183 wait complete. 2024-05-27T07:00:36.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 Processor -- start 2024-05-27T07:00:36.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- start start 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6900c60a0 0x7ff690155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6901f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6900ce6f0 con 0x7ff690072430 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff6900c7dd0 con 0x7ff6900c60a0 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7ff6900c5610 con 0x7ff69015a040 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34430/0 (socket says 172.21.15.81:34430) 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 -- 172.21.15.81:0/4168282761 learned_addr learned my addr 172.21.15.81:0/4168282761 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff686a00640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6900c60a0 0x7ff690155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68fe00640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6901f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 -- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 msgr2=0x7ff6901f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6901f9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 -- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6900c60a0 msgr2=0x7ff690155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6900c60a0 0x7ff690155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 -- 172.21.15.81:0/4168282761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff690161e80 con 0x7ff690072430 2024-05-27T07:00:36.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff68f400640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff68001a0d0 tx=0x7ff680019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800b0040 con 0x7ff690072430 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6800b9070 con 0x7ff690072430 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800b4640 con 0x7ff690072430 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff69014dba0 con 0x7ff690072430 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.660+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff69014c8a0 con 0x7ff690072430 2024-05-27T07:00:36.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff690162050 con 0x7ff690072430 2024-05-27T07:00:36.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7ff6800b3070 con 0x7ff690072430 2024-05-27T07:00:36.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff687e00640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 0x7ff66c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:36.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7ff6801286e0 con 0x7ff690072430 2024-05-27T07:00:36.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff686a00640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 0x7ff66c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:36.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.664+0000 7ff686a00640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 0x7ff66c071940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff6780096d0 tx=0x7ff678009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:36.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.668+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7ff6800c2020 con 0x7ff690072430 2024-05-27T07:00:36.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff69006a560 con 0x7ff690072430 2024-05-27T07:00:36.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff687e00640 1 -- 172.21.15.81:0/4168282761 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7ff680016090 con 0x7ff690072430 2024-05-27T07:00:36.849 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:36.849 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:36.849 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:36.850 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 msgr2=0x7ff66c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 0x7ff66c071940 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff6780096d0 tx=0x7ff678009240 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 msgr2=0x7ff69014d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff68001a0d0 tx=0x7ff680019240 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 shutdown_connections 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7ff69015a040 0x7ff6901f9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7ff66c06f430 0x7ff66c071940 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff6900c60a0 0x7ff690155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 --2- 172.21.15.81:0/4168282761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff690072430 0x7ff69014d660 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.860+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 >> 172.21.15.81:0/4168282761 conn(0x7ff690070700 msgr2=0x7ff6900c89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.864+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 shutdown_connections 2024-05-27T07:00:36.851 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:36.864+0000 7ff696800640 1 -- 172.21.15.81:0/4168282761 wait complete. 2024-05-27T07:00:36.870 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:38.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:39.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.040+0000 7f49a7800640 1 Processor -- start 2024-05-27T07:00:39.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.040+0000 7f49a7800640 1 -- start start 2024-05-27T07:00:39.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a7800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f49a006c790 0x7f49a006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a7800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a7800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a00c9f80 con 0x7f49a015cb80 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a7800640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f49a00c58b0 con 0x7f49a006c790 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a7800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f49a00c83f0 con 0x7f49a00c8880 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f499fe00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f49a006c790 0x7f49a006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:34824/0 (socket says 172.21.15.81:34824) 2024-05-27T07:00:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 -- 172.21.15.81:0/880272366 learned_addr learned my addr 172.21.15.81:0/880272366 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 -- 172.21.15.81:0/880272366 >> v2:172.21.15.106:3300/0 conn(0x7f49a006c790 msgr2=0x7f49a006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 --2- 172.21.15.81:0/880272366 >> v2:172.21.15.106:3300/0 conn(0x7f49a006c790 0x7f49a006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 -- 172.21.15.81:0/880272366 >> v1:172.21.15.106:6789/0 conn(0x7f49a015cb80 legacy=0x7f49a0152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.044+0000 7f49a5800640 1 -- 172.21.15.81:0/880272366 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a00c52f0 con 0x7f49a00c8880 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a5800640 1 --2- 172.21.15.81:0/880272366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4998009e50 tx=0x7f499801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9ffe317152f2537 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f499f400640 1 -- 172.21.15.81:0/880272366 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49980a5680 con 0x7f49a00c8880 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f499f400640 1 -- 172.21.15.81:0/880272366 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49980a5820 con 0x7f49a00c8880 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f499f400640 1 -- 172.21.15.81:0/880272366 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49980a5bf0 con 0x7f49a00c8880 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 -- 172.21.15.81:0/880272366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 msgr2=0x7f49a00c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 --2- 172.21.15.81:0/880272366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4998009e50 tx=0x7f499801ca30 comp rx=0 tx=0).stop 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 -- 172.21.15.81:0/880272366 shutdown_connections 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 --2- 172.21.15.81:0/880272366 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a00c8880 0x7f49a00c8c90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 --2- 172.21.15.81:0/880272366 >> v2:172.21.15.106:3300/0 conn(0x7f49a006c790 0x7f49a006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 -- 172.21.15.81:0/880272366 >> 172.21.15.81:0/880272366 conn(0x7f49a0070700 msgr2=0x7f49a00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:39.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 -- 172.21.15.81:0/880272366 shutdown_connections 2024-05-27T07:00:39.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.048+0000 7f49a7800640 1 -- 172.21.15.81:0/880272366 wait complete. 2024-05-27T07:00:39.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 Processor -- start 2024-05-27T07:00:39.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- start start 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49a006c790 0x7f49a0159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a015cb80 0x7f49a01429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f49a00c9c80 con 0x7f49a00c8880 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f49a00c9b00 con 0x7f49a006c790 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f49a01634e0 con 0x7f49a015cb80 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34442/0 (socket says 172.21.15.81:34442) 2024-05-27T07:00:39.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 -- 172.21.15.81:0/23853513 learned_addr learned my addr 172.21.15.81:0/23853513 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:39.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a4e00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49a006c790 0x7f49a0159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a5800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a015cb80 0x7f49a01429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 -- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a015cb80 msgr2=0x7f49a01429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a015cb80 0x7f49a01429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 -- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49a006c790 msgr2=0x7f49a0159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49a006c790 0x7f49a0159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 -- 172.21.15.81:0/23853513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a01616b0 con 0x7f49a00c8880 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499fe00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e870 tx=0x7f498c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c017070 con 0x7f49a00c8880 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f498c00c8e0 con 0x7f49a00c8880 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a015be50 con 0x7f49a00c8880 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c00ccd0 con 0x7f49a00c8880 2024-05-27T07:00:39.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.052+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a0142590 con 0x7f49a00c8880 2024-05-27T07:00:39.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.056+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a006a560 con 0x7f49a00c8880 2024-05-27T07:00:39.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.056+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f498c01a070 con 0x7f49a00c8880 2024-05-27T07:00:39.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.056+0000 7f499cc00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 0x7f4980071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:39.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.056+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f498c08f800 con 0x7f49a00c8880 2024-05-27T07:00:39.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.056+0000 7f49a4e00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 0x7f4980071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:39.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.060+0000 7f49a4e00640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 0x7f4980071910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4990007fd0 tx=0x7f4990009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:39.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.060+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f498c05bd00 con 0x7f49a00c8880 2024-05-27T07:00:39.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49a0149e90 con 0x7f49a00c8880 2024-05-27T07:00:39.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f499cc00640 1 -- 172.21.15.81:0/23853513 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f498c060000 con 0x7f49a00c8880 2024-05-27T07:00:39.244 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:39.244 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:39.244 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 msgr2=0x7f4980071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 0x7f4980071910 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4990007fd0 tx=0x7f4990009040 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 msgr2=0x7f49a015b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f498c00e870 tx=0x7f498c00ec70 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 shutdown_connections 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f49a015cb80 0x7f49a01429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f498006f400 0x7f4980071910 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c8880 0x7f49a015b910 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 --2- 172.21.15.81:0/23853513 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49a006c790 0x7f49a0159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 >> 172.21.15.81:0/23853513 conn(0x7f49a0070700 msgr2=0x7f49a00c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 shutdown_connections 2024-05-27T07:00:39.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:39.256+0000 7f49a7800640 1 -- 172.21.15.81:0/23853513 wait complete. 2024-05-27T07:00:39.265 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:39.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~275s 2024-05-27T07:00:41.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:41.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.436+0000 7fd71d400640 1 Processor -- start 2024-05-27T07:00:41.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.436+0000 7fd71d400640 1 -- start start 2024-05-27T07:00:41.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fd71815cb80 0x7fd718152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd71806cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd7180c8880 0x7fd7180c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd7180c58b0 con 0x7fd7180c8880 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fd7180c83f0 con 0x7fd71815cb80 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd71d400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7180c9f80 con 0x7fd71806c790 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd7180c8880 0x7fd7180c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd7180c8880 0x7fd7180c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34456/0 (socket says 172.21.15.81:34456) 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 -- 172.21.15.81:0/2936014043 learned_addr learned my addr 172.21.15.81:0/2936014043 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd716800640 1 --2- 172.21.15.81:0/2936014043 >> v2:172.21.15.106:3300/0 conn(0x7fd71815cb80 0x7fd718152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd715e00640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd71806cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 -- 172.21.15.81:0/2936014043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 msgr2=0x7fd71806cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:41.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd71806cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 -- 172.21.15.81:0/2936014043 >> v2:172.21.15.106:3300/0 conn(0x7fd71815cb80 msgr2=0x7fd718152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:41.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 --2- 172.21.15.81:0/2936014043 >> v2:172.21.15.106:3300/0 conn(0x7fd71815cb80 0x7fd718152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.440+0000 7fd717200640 1 -- 172.21.15.81:0/2936014043 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7180c52f0 con 0x7fd7180c8880 2024-05-27T07:00:41.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd717200640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd7180c8c90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd7180c8570 tx=0x7fd71001d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5fd2fabca97e5a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:41.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd715400640 1 -- 172.21.15.81:0/2936014043 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71000b040 con 0x7fd7180c8880 2024-05-27T07:00:41.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd715400640 1 -- 172.21.15.81:0/2936014043 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7100a7710 con 0x7fd7180c8880 2024-05-27T07:00:41.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd715400640 1 -- 172.21.15.81:0/2936014043 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7100a7ae0 con 0x7fd7180c8880 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 -- 172.21.15.81:0/2936014043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 msgr2=0x7fd7180c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd7180c8c90 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd7180c8570 tx=0x7fd71001d8f0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 -- 172.21.15.81:0/2936014043 shutdown_connections 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd7180c8c90 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 --2- 172.21.15.81:0/2936014043 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd71806cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 --2- 172.21.15.81:0/2936014043 >> v2:172.21.15.106:3300/0 conn(0x7fd71815cb80 0x7fd718152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 -- 172.21.15.81:0/2936014043 >> 172.21.15.81:0/2936014043 conn(0x7fd718070700 msgr2=0x7fd7180cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:41.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 -- 172.21.15.81:0/2936014043 shutdown_connections 2024-05-27T07:00:41.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.444+0000 7fd71d400640 1 -- 172.21.15.81:0/2936014043 wait complete. 2024-05-27T07:00:41.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 Processor -- start 2024-05-27T07:00:41.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- start start 2024-05-27T07:00:41.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd7181f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.437 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 0x7fd718147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd718153370 con 0x7fd7180c8880 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181531f0 con 0x7fd71815cb80 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7180c9b00 con 0x7fd71806c790 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34460/0 (socket says 172.21.15.81:34460) 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 -- 172.21.15.81:0/1505246996 learned_addr learned my addr 172.21.15.81:0/1505246996 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd717200640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 0x7fd718147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd716800640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd7181f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 -- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 msgr2=0x7fd7181f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:41.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd7181f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 -- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 msgr2=0x7fd718147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:41.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 0x7fd718147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 -- 172.21.15.81:0/1505246996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7181616b0 con 0x7fd7180c8880 2024-05-27T07:00:41.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd717200640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 0x7fd718147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:41.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd715e00640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd70400e840 tx=0x7fd70400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704017070 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70400c730 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71815a260 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70400cb20 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.448+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7181f6070 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.452+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71806a560 con 0x7fd7180c8880 2024-05-27T07:00:41.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.452+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd70401a070 con 0x7fd7180c8880 2024-05-27T07:00:41.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.452+0000 7fd70ea00640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 0x7fd6f8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:41.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.452+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd70408f760 con 0x7fd7180c8880 2024-05-27T07:00:41.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.452+0000 7fd716800640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 0x7fd6f8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:41.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.456+0000 7fd716800640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 0x7fd6f8071910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd7080073f0 tx=0x7fd70800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:41.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.456+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd70405c3c0 con 0x7fd7180c8880 2024-05-27T07:00:41.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7180cb210 con 0x7fd7180c8880 2024-05-27T07:00:41.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd70ea00640 1 -- 172.21.15.81:0/1505246996 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd70405bcc0 con 0x7fd7180c8880 2024-05-27T07:00:41.635 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:41.635 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:41.636 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 msgr2=0x7fd6f8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 0x7fd6f8071910 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd7080073f0 tx=0x7fd70800b040 comp rx=0 tx=0).stop 2024-05-27T07:00:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 msgr2=0x7fd718159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd70400e840 tx=0x7fd70400ed00 comp rx=0 tx=0).stop 2024-05-27T07:00:41.637 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 shutdown_connections 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd6f806f400 0x7fd6f8071910 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd71815cb80 0x7fd718147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7180c8880 0x7fd718159d20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 --2- 172.21.15.81:0/1505246996 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd71806c790 0x7fd7181f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 >> 172.21.15.81:0/1505246996 conn(0x7fd718070700 msgr2=0x7fd718074f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 shutdown_connections 2024-05-27T07:00:41.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:41.648+0000 7fd71d400640 1 -- 172.21.15.81:0/1505246996 wait complete. 2024-05-27T07:00:41.658 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:00:43.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:43.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.827+0000 7f4e06e00640 1 Processor -- start 2024-05-27T07:00:43.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.827+0000 7f4e06e00640 1 -- start start 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e06e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4e000c5f70 0x7f4e00165400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e06e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e00072840 con 0x7f4e00072430 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e06e00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e000c5270 con 0x7f4e000c5b60 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e06e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f4e000c5040 con 0x7f4e000c5f70 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e04e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4e000c5f70 0x7f4e00165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4dffe00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4e00072430 0x7f4e00159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45972/0 (socket says 172.21.15.81:45972) 2024-05-27T07:00:43.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4dffe00640 1 -- 172.21.15.81:0/707798456 learned_addr learned my addr 172.21.15.81:0/707798456 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e04e00640 1 -- 172.21.15.81:0/707798456 >> v1:172.21.15.106:6789/0 conn(0x7f4e000c5b60 legacy=0x7f4e00161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e04e00640 1 -- 172.21.15.81:0/707798456 >> v1:172.21.15.81:6789/0 conn(0x7f4e00072430 legacy=0x7f4e00159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4e04e00640 1 -- 172.21.15.81:0/707798456 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e000c7810 con 0x7f4e000c5f70 2024-05-27T07:00:43.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.831+0000 7f4dff400640 1 -- 172.21.15.81:0/707798456 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116994971 0 0) 0x7f4e00072840 con 0x7f4e00072430 2024-05-27T07:00:43.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e04e00640 1 --2- 172.21.15.81:0/707798456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5f70 0x7f4e00165400 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6238aaa5e3b5b155 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:43.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4dff400640 1 -- 172.21.15.81:0/707798456 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80b0070 con 0x7f4e000c5f70 2024-05-27T07:00:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4dff400640 1 -- 172.21.15.81:0/707798456 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df80ab600 con 0x7f4e000c5f70 2024-05-27T07:00:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4dff400640 1 -- 172.21.15.81:0/707798456 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80ab9d0 con 0x7f4e000c5f70 2024-05-27T07:00:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 -- 172.21.15.81:0/707798456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5f70 msgr2=0x7f4e00165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 --2- 172.21.15.81:0/707798456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5f70 0x7f4e00165400 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4df80097c0 tx=0x7f4df801c4a0 comp rx=0 tx=0).stop 2024-05-27T07:00:43.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 -- 172.21.15.81:0/707798456 shutdown_connections 2024-05-27T07:00:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 --2- 172.21.15.81:0/707798456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5f70 0x7f4e00165400 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 -- 172.21.15.81:0/707798456 >> 172.21.15.81:0/707798456 conn(0x7f4e00070700 msgr2=0x7f4e000cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 -- 172.21.15.81:0/707798456 shutdown_connections 2024-05-27T07:00:43.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.835+0000 7f4e06e00640 1 -- 172.21.15.81:0/707798456 wait complete. 2024-05-27T07:00:43.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 Processor -- start 2024-05-27T07:00:43.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- start start 2024-05-27T07:00:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e00072430 0x7f4e00154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:43.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 0x7f4e0014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e000c7dd0 con 0x7f4e000c5b60 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4e000c75c0 con 0x7f4e000c5f70 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4e000ce6f0 con 0x7f4e00072430 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34486/0 (socket says 172.21.15.81:34486) 2024-05-27T07:00:43.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 -- 172.21.15.81:0/542913240 learned_addr learned my addr 172.21.15.81:0/542913240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e04e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 0x7f4e0014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4dffe00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e00072430 0x7f4e00154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 -- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e00072430 msgr2=0x7f4e00154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e00072430 0x7f4e00154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:43.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 -- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 msgr2=0x7f4e0014a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 0x7f4e0014a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 -- 172.21.15.81:0/542913240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e00154fb0 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e04e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 0x7f4e0014a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4df7400640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c460 tx=0x7f4de800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de80052d0 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de8005470 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8012870 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e000ccf10 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.839+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e00155860 con 0x7f4e000c5b60 2024-05-27T07:00:43.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0006a560 con 0x7f4e000c5b60 2024-05-27T07:00:43.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4de8012a10 con 0x7f4e000c5b60 2024-05-27T07:00:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4dfcc00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 0x7f4ddc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4de808f160 con 0x7f4e000c5b60 2024-05-27T07:00:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4dffe00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 0x7f4ddc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:43.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.843+0000 7f4dffe00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 0x7f4ddc071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4df0005fd0 tx=0x7f4df0005c20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:43.838 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:43.847+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4de805bd30 con 0x7f4e000c5b60 2024-05-27T07:00:44.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.039+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e000c5940 con 0x7f4e000c5b60 2024-05-27T07:00:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.039+0000 7f4dfcc00640 1 -- 172.21.15.81:0/542913240 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4de805b630 con 0x7f4e000c5b60 2024-05-27T07:00:44.030 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:44.030 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:44.030 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 msgr2=0x7f4ddc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 0x7f4ddc071910 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4df0005fd0 tx=0x7f4df0005c20 comp rx=0 tx=0).stop 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 msgr2=0x7f4e000cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c460 tx=0x7f4de800c920 comp rx=0 tx=0).stop 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 shutdown_connections 2024-05-27T07:00:44.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4ddc06f400 0x7f4ddc071910 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e000c5f70 0x7f4e0014a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e000c5b60 0x7f4e000cc360 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 --2- 172.21.15.81:0/542913240 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e00072430 0x7f4e00154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 >> 172.21.15.81:0/542913240 conn(0x7f4e00070700 msgr2=0x7f4e00075de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 shutdown_connections 2024-05-27T07:00:44.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:44.043+0000 7f4e06e00640 1 -- 172.21.15.81:0/542913240 wait complete. 2024-05-27T07:00:44.052 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:46.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:46.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.223+0000 7f8337800640 1 Processor -- start 2024-05-27T07:00:46.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.223+0000 7f8337800640 1 -- start start 2024-05-27T07:00:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8337800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f83300c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8337800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8330072840 con 0x7f8330072430 2024-05-27T07:00:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8337800640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83300c5270 con 0x7f8330161cd0 2024-05-27T07:00:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8337800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f83300c5040 con 0x7f83300c5b60 2024-05-27T07:00:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8334e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8330072430 0x7f8330159b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47394/0 (socket says 172.21.15.81:47394) 2024-05-27T07:00:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8334e00640 1 -- 172.21.15.81:0/68438657 learned_addr learned my addr 172.21.15.81:0/68438657 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8327e00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f83300c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539588959 0 0) 0x7f8330072840 con 0x7f8330072430 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8314005740 con 0x7f8330072430 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4245504980 0 0) 0x7f83300c5270 con 0x7f8330161cd0 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8330072840 con 0x7f8330161cd0 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2075822271 0 0) 0x7f8314005740 con 0x7f8330072430 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 msgr2=0x7f83300c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f83300c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 >> v1:172.21.15.106:6789/0 conn(0x7f8330161cd0 legacy=0x7f83301640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83300c7810 con 0x7f8330072430 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f8320001700 con 0x7f8330072430 2024-05-27T07:00:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 >> v1:172.21.15.81:6789/0 conn(0x7f8330072430 legacy=0x7f8330159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f8314005e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83140063e0 0x7f8314006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8314006fb0 0x7f8314007440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f831400de10 con 0x7f8314006fb0 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f831400dfc0 con 0x7f83140063e0 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f831400e1c0 con 0x7f83300c5b60 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8314006fb0 0x7f8314007440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8334e00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f8314005e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8327e00640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83140063e0 0x7f8314006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 -- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 msgr2=0x7f8314005e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f83300c5b60 0x7f8314005e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 -- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83140063e0 msgr2=0x7f8314006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:46.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f83140063e0 0x7f8314006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.227+0000 7f8335800640 1 -- 172.21.15.81:0/68438657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8314005740 con 0x7f8314006fb0 2024-05-27T07:00:46.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8335800640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8314006fb0 0x7f8314007440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8328002080 tx=0x7f832801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7281a20aef6635e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:46.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328001c50 con 0x7f8314006fb0 2024-05-27T07:00:46.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8328006d60 con 0x7f8314006fb0 2024-05-27T07:00:46.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f832fe00640 1 -- 172.21.15.81:0/68438657 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328003bc0 con 0x7f8314006fb0 2024-05-27T07:00:46.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 -- 172.21.15.81:0/68438657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8314006fb0 msgr2=0x7f8314007440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 --2- 172.21.15.81:0/68438657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8314006fb0 0x7f8314007440 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8328002080 tx=0x7f832801c900 comp rx=0 tx=0).stop 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8335800640 1 -- 172.21.15.81:0/68438657 reap_dead start 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 -- 172.21.15.81:0/68438657 shutdown_connections 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 -- 172.21.15.81:0/68438657 >> 172.21.15.81:0/68438657 conn(0x7f8330070700 msgr2=0x7f83300cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 -- 172.21.15.81:0/68438657 shutdown_connections 2024-05-27T07:00:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.231+0000 7f8337800640 1 -- 172.21.15.81:0/68438657 wait complete. 2024-05-27T07:00:46.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 Processor -- start 2024-05-27T07:00:46.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 -- start start 2024-05-27T07:00:46.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 0x7f8330144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 0x7f8330073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83300c7dd0 con 0x7f833014dcc0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f83300c75c0 con 0x7f833014d6d0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8337800640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f83301595e0 con 0x7f8314006fb0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37040/0 (socket says 172.21.15.81:37040) 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 -- 172.21.15.81:0/1396317877 learned_addr learned my addr 172.21.15.81:0/1396317877 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8334e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 0x7f8330144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8335800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 0x7f8330073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 -- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 msgr2=0x7f8330144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 0x7f8330144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 -- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 msgr2=0x7f8330073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 0x7f8330073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 -- 172.21.15.81:0/1396317877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833014c870 con 0x7f833014dcc0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8335800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 0x7f8330073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8334e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 0x7f8330144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f8327e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8318011970 tx=0x7f8318011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.235+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8318019070 con 0x7f833014dcc0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8318004240 con 0x7f833014dcc0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831800f8e0 con 0x7f833014dcc0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83301f2c10 con 0x7f833014dcc0 2024-05-27T07:00:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83301f31d0 con 0x7f833014dcc0 2024-05-27T07:00:46.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83301f2970 con 0x7f833014dcc0 2024-05-27T07:00:46.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f831801c070 con 0x7f833014dcc0 2024-05-27T07:00:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f832d600640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 0x7f830c071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f831808ff40 con 0x7f833014dcc0 2024-05-27T07:00:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.239+0000 7f8334e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 0x7f830c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.243+0000 7f8334e00640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 0x7f830c071940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8320001060 tx=0x7f8320009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.243+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f83301f2970 con 0x7f833014dcc0 2024-05-27T07:00:46.300 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~282s 2024-05-27T07:00:46.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.431+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f833006a560 con 0x7f833014dcc0 2024-05-27T07:00:46.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.431+0000 7f832d600640 1 -- 172.21.15.81:0/1396317877 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f831805cb90 con 0x7f833014dcc0 2024-05-27T07:00:46.422 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:46.422 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:46.423 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:46.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 msgr2=0x7f830c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 0x7f830c071940 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8320001060 tx=0x7f8320009040 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 msgr2=0x7f833014c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f8318011970 tx=0x7f8318011e30 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 shutdown_connections 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f830c06f430 0x7f830c071940 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f833014d6d0 0x7f8330073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833014dcc0 0x7f833014c330 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 --2- 172.21.15.81:0/1396317877 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f8314006fb0 0x7f8330144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 >> 172.21.15.81:0/1396317877 conn(0x7f8330070700 msgr2=0x7f833014f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 shutdown_connections 2024-05-27T07:00:46.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:46.435+0000 7f8337800640 1 -- 172.21.15.81:0/1396317877 wait complete. 2024-05-27T07:00:46.445 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:48.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 Processor -- start 2024-05-27T07:00:48.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 -- start start 2024-05-27T07:00:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc0cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39cc0c78b0 con 0x7f39cc0cbd00 2024-05-27T07:00:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f39cc154b50 con 0x7f39cc154200 2024-05-27T07:00:48.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39d1a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc0c6ee0 con 0x7f39cc0ca770 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39cb400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39caa00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc0cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39cb400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:34978/0 (socket says 172.21.15.81:34978) 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39cb400640 1 -- 172.21.15.81:0/1078136955 learned_addr learned my addr 172.21.15.81:0/1078136955 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39ca000640 1 --1- 172.21.15.81:0/1078136955 >> v1:172.21.15.106:6789/0 conn(0x7f39cc0cbd00 0x7f39cc0cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:49410/0 (socket says 172.21.15.81:49410) 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39caa00640 1 -- 172.21.15.81:0/1078136955 >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 msgr2=0x7f39cc154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39caa00640 1 --2- 172.21.15.81:0/1078136955 >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39caa00640 1 -- 172.21.15.81:0/1078136955 >> v1:172.21.15.106:6789/0 conn(0x7f39cc0cbd00 legacy=0x7f39cc0cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39caa00640 1 -- 172.21.15.81:0/1078136955 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc0c71c0 con 0x7f39cc0ca770 2024-05-27T07:00:48.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.619+0000 7f39cb400640 1 --2- 172.21.15.81:0/1078136955 >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39caa00640 1 --2- 172.21.15.81:0/1078136955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc0cb7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009d90 tx=0x7f39bc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36aec69b3e29827 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39c9600640 1 -- 172.21.15.81:0/1078136955 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0b2070 con 0x7f39cc0ca770 2024-05-27T07:00:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39c9600640 1 -- 172.21.15.81:0/1078136955 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39bc0a5940 con 0x7f39cc0ca770 2024-05-27T07:00:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39c9600640 1 -- 172.21.15.81:0/1078136955 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc0a5d10 con 0x7f39cc0ca770 2024-05-27T07:00:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39d1a00640 1 -- 172.21.15.81:0/1078136955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 msgr2=0x7f39cc0cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.623+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1078136955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc0cb7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009d90 tx=0x7f39bc01c970 comp rx=0 tx=0).stop 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- 172.21.15.81:0/1078136955 shutdown_connections 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1078136955 >> v2:172.21.15.106:3300/0 conn(0x7f39cc154200 0x7f39cc154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1078136955 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc0cb7c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- 172.21.15.81:0/1078136955 >> 172.21.15.81:0/1078136955 conn(0x7f39cc068ae0 msgr2=0x7f39cc068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- 172.21.15.81:0/1078136955 shutdown_connections 2024-05-27T07:00:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- 172.21.15.81:0/1078136955 wait complete. 2024-05-27T07:00:48.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 Processor -- start 2024-05-27T07:00:48.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- start start 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39cc0cbd00 0x7f39cc150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc0d0640 con 0x7f39cc154200 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc0d04c0 con 0x7f39cc0cbd00 2024-05-27T07:00:48.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.627+0000 7f39d1a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc167590 con 0x7f39cc0ca770 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37054/0 (socket says 172.21.15.81:37054) 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 -- 172.21.15.81:0/1593296286 learned_addr learned my addr 172.21.15.81:0/1593296286 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39caa00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39ca000640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39cc0cbd00 0x7f39cc150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 -- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 msgr2=0x7f39cc06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 -- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39cc0cbd00 msgr2=0x7f39cc150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39cc0cbd00 0x7f39cc150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 -- 172.21.15.81:0/1593296286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc1f9a60 con 0x7f39cc154200 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39cb400640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e930 tx=0x7f39c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4017070 con 0x7f39cc154200 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c4004020 con 0x7f39cc154200 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c40043f0 con 0x7f39cc154200 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc14e410 con 0x7f39cc154200 2024-05-27T07:00:48.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc06f1b0 con 0x7f39cc154200 2024-05-27T07:00:48.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc06a560 con 0x7f39cc154200 2024-05-27T07:00:48.622 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.631+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f39c400c940 con 0x7f39cc154200 2024-05-27T07:00:48.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.635+0000 7f39c2a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 0x7f39ac071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:48.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.635+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f39c408f320 con 0x7f39cc154200 2024-05-27T07:00:48.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.635+0000 7f39caa00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 0x7f39ac071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:48.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.635+0000 7f39caa00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 0x7f39ac071940 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0af000 tx=0x7f39bc0a7070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:48.628 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.639+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f39c405b8a0 con 0x7f39cc154200 2024-05-27T07:00:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.827+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39cc0caec0 con 0x7f39cc154200 2024-05-27T07:00:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39c2a00640 1 -- 172.21.15.81:0/1593296286 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f39c405fba0 con 0x7f39cc154200 2024-05-27T07:00:48.819 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:48.819 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:48.819 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 msgr2=0x7f39ac071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 0x7f39ac071940 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0af000 tx=0x7f39bc0a7070 comp rx=0 tx=0).stop 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 msgr2=0x7f39cc145080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f39c400e930 tx=0x7f39c400edf0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 shutdown_connections 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f39ac06f430 0x7f39ac071940 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc154200 0x7f39cc145080 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39cc0cbd00 0x7f39cc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 --2- 172.21.15.81:0/1593296286 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f39cc0ca770 0x7f39cc06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:48.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 >> 172.21.15.81:0/1593296286 conn(0x7f39cc068ae0 msgr2=0x7f39cc152280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:48.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 shutdown_connections 2024-05-27T07:00:48.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:48.831+0000 7f39d1a00640 1 -- 172.21.15.81:0/1593296286 wait complete. 2024-05-27T07:00:48.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:50.841 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 Processor -- start 2024-05-27T07:00:51.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 -- start start 2024-05-27T07:00:51.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4e9c072430 0x7f4e9c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f4e9c161cd0 0x7f4e9c1640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e9c072840 con 0x7f4e9c15a040 2024-05-27T07:00:51.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9c0c5040 con 0x7f4e9c072430 2024-05-27T07:00:51.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.011+0000 7f4ea1000640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9c0c75c0 con 0x7f4e9c161cd0 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4e9c072430 0x7f4e9c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e91400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4e9c15a040 0x7f4e9c0c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47404/0 (socket says 172.21.15.81:47404) 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e91400640 1 -- 172.21.15.81:0/1717374027 learned_addr learned my addr 172.21.15.81:0/1717374027 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e9a800640 1 --2- 172.21.15.81:0/1717374027 >> v2:172.21.15.106:3300/0 conn(0x7f4e9c161cd0 0x7f4e9c1640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 -- 172.21.15.81:0/1717374027 >> v2:172.21.15.106:3300/0 conn(0x7f4e9c161cd0 msgr2=0x7f4e9c1640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 --2- 172.21.15.81:0/1717374027 >> v2:172.21.15.106:3300/0 conn(0x7f4e9c161cd0 0x7f4e9c1640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 -- 172.21.15.81:0/1717374027 >> v1:172.21.15.81:6789/0 conn(0x7f4e9c15a040 legacy=0x7f4e9c0c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 -- 172.21.15.81:0/1717374027 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c0c5270 con 0x7f4e9c072430 2024-05-27T07:00:51.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99400640 1 -- 172.21.15.81:0/1717374027 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3564967999 0 0) 0x7f4e9c072840 con 0x7f4e9c15a040 2024-05-27T07:00:51.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99e00640 1 --2- 172.21.15.81:0/1717374027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c072430 0x7f4e9c159b00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0c9d90 tx=0x7f4e8c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2095a714ffc2afcb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:51.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99400640 1 -- 172.21.15.81:0/1717374027 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0b0070 con 0x7f4e9c072430 2024-05-27T07:00:51.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99400640 1 -- 172.21.15.81:0/1717374027 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c0ab600 con 0x7f4e9c072430 2024-05-27T07:00:51.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.015+0000 7f4e99400640 1 -- 172.21.15.81:0/1717374027 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c0ab9d0 con 0x7f4e9c072430 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- 172.21.15.81:0/1717374027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c072430 msgr2=0x7f4e9c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 --2- 172.21.15.81:0/1717374027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c072430 0x7f4e9c159b00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0c9d90 tx=0x7f4e8c01c480 comp rx=0 tx=0).stop 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- 172.21.15.81:0/1717374027 shutdown_connections 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 --2- 172.21.15.81:0/1717374027 >> v2:172.21.15.106:3300/0 conn(0x7f4e9c161cd0 0x7f4e9c1640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 --2- 172.21.15.81:0/1717374027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c072430 0x7f4e9c159b00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- 172.21.15.81:0/1717374027 >> 172.21.15.81:0/1717374027 conn(0x7f4e9c070700 msgr2=0x7f4e9c0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- 172.21.15.81:0/1717374027 shutdown_connections 2024-05-27T07:00:51.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- 172.21.15.81:0/1717374027 wait complete. 2024-05-27T07:00:51.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 Processor -- start 2024-05-27T07:00:51.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.019+0000 7f4ea1000640 1 -- start start 2024-05-27T07:00:51.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e9c072430 0x7f4e9c1666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e9c161cd0 0x7f4e9c14da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c0ce6f0 con 0x7f4e9c15a040 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c0c7dd0 con 0x7f4e9c161cd0 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c0c5610 con 0x7f4e9c072430 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37080/0 (socket says 172.21.15.81:37080) 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 -- 172.21.15.81:0/3723523463 learned_addr learned my addr 172.21.15.81:0/3723523463 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e9a800640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e9c161cd0 0x7f4e9c14da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e99e00640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e9c072430 0x7f4e9c1666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 -- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e9c072430 msgr2=0x7f4e9c1666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e9c072430 0x7f4e9c1666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 -- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e9c161cd0 msgr2=0x7f4e9c14da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e9c161cd0 0x7f4e9c14da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 -- 172.21.15.81:0/3723523463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c1f94f0 con 0x7f4e9c15a040 2024-05-27T07:00:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e91400640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400c430 tx=0x7f4e8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e840074e0 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e840047f0 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84007680 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c167030 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c158350 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c166da0 con 0x7f4e9c15a040 2024-05-27T07:00:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.023+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4e84011070 con 0x7f4e9c15a040 2024-05-27T07:00:51.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.027+0000 7f4e92a00640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 0x7f4e78071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:51.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.027+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f4e8408f100 con 0x7f4e9c15a040 2024-05-27T07:00:51.015 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.027+0000 7f4e99e00640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 0x7f4e78071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:51.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.027+0000 7f4e99e00640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 0x7f4e78071910 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0ad000 tx=0x7f4e8c0a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:51.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.031+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f4e8405bd50 con 0x7f4e9c15a040 2024-05-27T07:00:51.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.219+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e9c06a560 con 0x7f4e9c15a040 2024-05-27T07:00:51.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.219+0000 7f4e92a00640 1 -- 172.21.15.81:0/3723523463 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f4e8405b650 con 0x7f4e9c15a040 2024-05-27T07:00:51.210 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:51.210 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:51.210 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 msgr2=0x7f4e78071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 0x7f4e78071910 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0ad000 tx=0x7f4e8c0a6000 comp rx=0 tx=0).stop 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 msgr2=0x7f4e9c1f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400c430 tx=0x7f4e8400c8f0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 shutdown_connections 2024-05-27T07:00:51.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f4e7806f400 0x7f4e78071910 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4e9c161cd0 0x7f4e9c14da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c15a040 0x7f4e9c1f8fb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 --2- 172.21.15.81:0/3723523463 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4e9c072430 0x7f4e9c1666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 >> 172.21.15.81:0/3723523463 conn(0x7f4e9c070700 msgr2=0x7f4e9c073e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 shutdown_connections 2024-05-27T07:00:51.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:51.223+0000 7f4ea1000640 1 -- 172.21.15.81:0/3723523463 wait complete. 2024-05-27T07:00:51.232 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:53.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~289s 2024-05-27T07:00:53.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.403+0000 7f24a2a00640 1 Processor -- start 2024-05-27T07:00:53.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.403+0000 7f24a2a00640 1 -- start start 2024-05-27T07:00:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a2a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f249c1606c0 0x7f249c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a2a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c15c850 con 0x7f249c06e290 2024-05-27T07:00:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a2a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f249c06e740 con 0x7f249c0cc8c0 2024-05-27T07:00:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a2a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f249c15caf0 con 0x7f249c1606c0 2024-05-27T07:00:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f249c06e290 0x7f249c156920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47418/0 (socket says 172.21.15.81:47418) 2024-05-27T07:00:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 -- 172.21.15.81:0/3473894124 learned_addr learned my addr 172.21.15.81:0/3473894124 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f2493400640 1 --2- 172.21.15.81:0/3473894124 >> v2:172.21.15.106:3300/0 conn(0x7f249c1606c0 0x7f249c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054686881 0 0) 0x7f249c15c850 con 0x7f249c06e290 2024-05-27T07:00:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2480003530 con 0x7f249c06e290 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.1 v1:172.21.15.106:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771153224 0 0) 0x7f249c06e740 con 0x7f249c0cc8c0 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> v1:172.21.15.106:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f249c15c850 con 0x7f249c0cc8c0 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1487209318 0 0) 0x7f2480003530 con 0x7f249c06e290 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 >> v2:172.21.15.106:3300/0 conn(0x7f249c1606c0 msgr2=0x7f249c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 --2- 172.21.15.81:0/3473894124 >> v2:172.21.15.106:3300/0 conn(0x7f249c1606c0 0x7f249c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 >> v1:172.21.15.106:6789/0 conn(0x7f249c0cc8c0 legacy=0x7f249c166fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c15cdd0 con 0x7f249c06e290 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f248c001720 con 0x7f249c06e290 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 >> v1:172.21.15.81:6789/0 conn(0x7f249c06e290 legacy=0x7f249c156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f2480004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2480004810 0x7f2480004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480005290 0x7f248000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f249c15caf0 con 0x7f2480003c40 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f248000c1a0 con 0x7f2480005290 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f248000c3a0 con 0x7f2480004810 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f2480004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a0a00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480005290 0x7f248000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f2493400640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2480004810 0x7f2480004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 -- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2480004810 msgr2=0x7f2480004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f2480004810 0x7f2480004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 -- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480005290 msgr2=0x7f248000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2480005290 0x7f248000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f249be00640 1 -- 172.21.15.81:0/3473894124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2480003530 con 0x7f2480003c40 2024-05-27T07:00:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.407+0000 7f24a0a00640 1 -- 172.21.15.81:0/3473894124 reap_dead start 2024-05-27T07:00:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f249be00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f2480004090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c160 tx=0x7f248c01cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe31a55e61fcffb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c007cb0 con 0x7f2480003c40 2024-05-27T07:00:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c0a9020 con 0x7f2480003c40 2024-05-27T07:00:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f249b400640 1 -- 172.21.15.81:0/3473894124 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c005b00 con 0x7f2480003c40 2024-05-27T07:00:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 -- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 msgr2=0x7f2480004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f2480004090 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c160 tx=0x7f248c01cae0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 -- 172.21.15.81:0/3473894124 shutdown_connections 2024-05-27T07:00:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 --2- 172.21.15.81:0/3473894124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f2480004090 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 -- 172.21.15.81:0/3473894124 >> 172.21.15.81:0/3473894124 conn(0x7f249c068ab0 msgr2=0x7f249c068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 -- 172.21.15.81:0/3473894124 shutdown_connections 2024-05-27T07:00:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.411+0000 7f24a2a00640 1 -- 172.21.15.81:0/3473894124 wait complete. 2024-05-27T07:00:53.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 Processor -- start 2024-05-27T07:00:53.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- start start 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f249c0d0370 0x7f249c151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249c0cf240 0x7f249c150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f249c160ad0 con 0x7f2480003c40 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f249c15d5d0 con 0x7f249c0d0370 2024-05-27T07:00:53.405 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f249c06ea20 con 0x7f249c0cf240 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37100/0 (socket says 172.21.15.81:37100) 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 -- 172.21.15.81:0/1175136250 learned_addr learned my addr 172.21.15.81:0/1175136250 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a0a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249c0cf240 0x7f249c150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f2493400640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f249c0d0370 0x7f249c151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 -- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249c0cf240 msgr2=0x7f249c150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249c0cf240 0x7f249c150440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 -- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f249c0d0370 msgr2=0x7f249c151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f249c0d0370 0x7f249c151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 -- 172.21.15.81:0/1175136250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c1f3400 con 0x7f2480003c40 2024-05-27T07:00:53.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f249be00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f248c0ab040 tx=0x7f2480005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:53.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c007cb0 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c0b2070 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c001c10 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c1f29c0 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.415+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c0cdc10 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249c06a560 con 0x7f2480003c40 2024-05-27T07:00:53.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f248c007510 con 0x7f2480003c40 2024-05-27T07:00:53.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f2498c00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 0x7f2478071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:53.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f248c1289f0 con 0x7f2480003c40 2024-05-27T07:00:53.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f2493400640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 0x7f2478071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:53.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.419+0000 7f2493400640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 0x7f2478071940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f249c15ec90 tx=0x7f2484008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:53.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.423+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f248c0be030 con 0x7f2480003c40 2024-05-27T07:00:53.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.615+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f249c06de60 con 0x7f2480003c40 2024-05-27T07:00:53.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.615+0000 7f2498c00640 1 -- 172.21.15.81:0/1175136250 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f248c016890 con 0x7f2480003c40 2024-05-27T07:00:53.606 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:53.606 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:53.607 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:53.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 msgr2=0x7f2478071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.608 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 0x7f2478071940 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f249c15ec90 tx=0x7f2484008040 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 msgr2=0x7f249c1f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f248c0ab040 tx=0x7f2480005840 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 shutdown_connections 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f249c0cf240 0x7f249c150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f247806f430 0x7f2478071940 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f249c0d0370 0x7f249c151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 --2- 172.21.15.81:0/1175136250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2480003c40 0x7f249c1f2ec0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:53.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 >> 172.21.15.81:0/1175136250 conn(0x7f249c068ab0 msgr2=0x7f249c154650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:53.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 shutdown_connections 2024-05-27T07:00:53.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:53.619+0000 7f24a2a00640 1 -- 172.21.15.81:0/1175136250 wait complete. 2024-05-27T07:00:53.629 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:00:55.630 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:55.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 Processor -- start 2024-05-27T07:00:55.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 -- start start 2024-05-27T07:00:55.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 0x7f48781606c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f48780cc8c0 0x7f48780cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48780caf50 con 0x7f4878156920 2024-05-27T07:00:55.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f4878160c00 con 0x7f48780cc8c0 2024-05-27T07:00:55.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f487ca00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f487815c850 con 0x7f487806e290 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4875400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4878156920 0x7f48780ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35342/0 (socket says 172.21.15.81:35342) 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4875400640 1 -- 172.21.15.81:0/1732150912 learned_addr learned my addr 172.21.15.81:0/1732150912 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4876800640 1 --2- 172.21.15.81:0/1732150912 >> v2:172.21.15.106:3300/0 conn(0x7f48780cc8c0 0x7f48780cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4875e00640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 0x7f48781606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894632244 0 0) 0x7f48780caf50 con 0x7f4878156920 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f485c003530 con 0x7f4878156920 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1682086490 0 0) 0x7f485c003530 con 0x7f4878156920 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4876800640 1 -- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 msgr2=0x7f48781606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4876800640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 0x7f48781606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:55.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4876800640 1 -- 172.21.15.81:0/1732150912 >> v1:172.21.15.81:6789/0 conn(0x7f4878156920 legacy=0x7f48780ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:55.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4876800640 1 -- 172.21.15.81:0/1732150912 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487806e740 con 0x7f48780cc8c0 2024-05-27T07:00:55.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.803+0000 7f4875e00640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 0x7f48781606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:00:55.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f4876800640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48780cc8c0 0x7f48780cccd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4870009d90 tx=0x7f487001f080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=350097221f82cf7e server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:55.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48700b5070 con 0x7f48780cc8c0 2024-05-27T07:00:55.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48700a8bd0 con 0x7f48780cc8c0 2024-05-27T07:00:55.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f4874a00640 1 -- 172.21.15.81:0/1732150912 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864001700 con 0x7f48780cc8c0 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 -- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48780cc8c0 msgr2=0x7f48780cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48780cc8c0 0x7f48780cccd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4870009d90 tx=0x7f487001f080 comp rx=0 tx=0).stop 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 -- 172.21.15.81:0/1732150912 shutdown_connections 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48780cc8c0 0x7f48780cccd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 --2- 172.21.15.81:0/1732150912 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f487806e290 0x7f48781606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 -- 172.21.15.81:0/1732150912 >> 172.21.15.81:0/1732150912 conn(0x7f4878068ab0 msgr2=0x7f4878068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:55.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 -- 172.21.15.81:0/1732150912 shutdown_connections 2024-05-27T07:00:55.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.807+0000 7f487ca00640 1 -- 172.21.15.81:0/1732150912 wait complete. 2024-05-27T07:00:55.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 Processor -- start 2024-05-27T07:00:55.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 -- start start 2024-05-27T07:00:55.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f487806e290 0x7f48780c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4878156920 0x7f48780cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f487815d0b0 con 0x7f48780cc8c0 2024-05-27T07:00:55.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f487815cdd0 con 0x7f487806e290 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f487ca00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f487815d5d0 con 0x7f4878156920 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59220/0 (socket says 172.21.15.81:59220) 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 -- 172.21.15.81:0/779797172 learned_addr learned my addr 172.21.15.81:0/779797172 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4876800640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4878156920 0x7f48780cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875e00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f487806e290 0x7f48780c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 -- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4878156920 msgr2=0x7f48780cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4878156920 0x7f48780cfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 -- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f487806e290 msgr2=0x7f48780c8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f487806e290 0x7f48780c8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.811+0000 7f4875400640 1 -- 172.21.15.81:0/779797172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48781f2cd0 con 0x7f48780cc8c0 2024-05-27T07:00:55.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f4875400640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f486400c4e0 tx=0x7f486400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:55.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864005cf0 con 0x7f48780cc8c0 2024-05-27T07:00:55.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4864007070 con 0x7f48780cc8c0 2024-05-27T07:00:55.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864007460 con 0x7f48780cc8c0 2024-05-27T07:00:55.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487815fdb0 con 0x7f48780cc8c0 2024-05-27T07:00:55.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48780c86d0 con 0x7f48780cc8c0 2024-05-27T07:00:55.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487806a560 con 0x7f48780cc8c0 2024-05-27T07:00:55.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.815+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f4864007600 con 0x7f48780cc8c0 2024-05-27T07:00:55.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.819+0000 7f486e000640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 0x7f4858071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:55.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.819+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f486408f310 con 0x7f48780cc8c0 2024-05-27T07:00:55.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.819+0000 7f4875e00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 0x7f4858071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:55.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.819+0000 7f4875e00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 0x7f4858071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4868009a50 tx=0x7f4868008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:55.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:55.823+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f486405bf70 con 0x7f48780cc8c0 2024-05-27T07:00:55.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f487806de60 con 0x7f48780cc8c0 2024-05-27T07:00:56.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f486e000640 1 -- 172.21.15.81:0/779797172 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f486405b870 con 0x7f48780cc8c0 2024-05-27T07:00:56.000 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:56.000 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:56.001 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:56.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 msgr2=0x7f4858071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:56.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 0x7f4858071910 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4868009a50 tx=0x7f4868008040 comp rx=0 tx=0).stop 2024-05-27T07:00:56.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 msgr2=0x7f487815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f486400c4e0 tx=0x7f486400c9a0 comp rx=0 tx=0).stop 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 shutdown_connections 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f4878156920 0x7f48780cfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f485806f400 0x7f4858071910 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48780cc8c0 0x7f487815e820 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 --2- 172.21.15.81:0/779797172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f487806e290 0x7f48780c8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 >> 172.21.15.81:0/779797172 conn(0x7f4878068ab0 msgr2=0x7f4878152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 shutdown_connections 2024-05-27T07:00:56.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:56.011+0000 7f487ca00640 1 -- 172.21.15.81:0/779797172 wait complete. 2024-05-27T07:00:56.022 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:58.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:00:58.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 Processor -- start 2024-05-27T07:00:58.183 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 -- start start 2024-05-27T07:00:58.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6f240ca770 0x7f6f240cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6f24154200 0x7f6f24154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f240c78b0 con 0x7f6f240cbd00 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f6f24154b50 con 0x7f6f24154200 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f29400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f6f240c6ee0 con 0x7f6f240ca770 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f23200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6f24154200 0x7f6f24154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f21e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6f240cbd00 0x7f6f240cfdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35344/0 (socket says 172.21.15.81:35344) 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f21e00640 1 -- 172.21.15.81:0/3232926907 learned_addr learned my addr 172.21.15.81:0/3232926907 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:00:58.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f22800640 1 --2- 172.21.15.81:0/3232926907 >> v2:172.21.15.106:3300/0 conn(0x7f6f240ca770 0x7f6f240cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f23200640 1 -- 172.21.15.81:0/3232926907 >> v2:172.21.15.106:3300/0 conn(0x7f6f240ca770 msgr2=0x7f6f240cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:00:58.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f23200640 1 --2- 172.21.15.81:0/3232926907 >> v2:172.21.15.106:3300/0 conn(0x7f6f240ca770 0x7f6f240cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f23200640 1 -- 172.21.15.81:0/3232926907 >> v1:172.21.15.81:6789/0 conn(0x7f6f240cbd00 legacy=0x7f6f240cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f23200640 1 -- 172.21.15.81:0/3232926907 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f240c71c0 con 0x7f6f24154200 2024-05-27T07:00:58.186 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.195+0000 7f6f21400640 1 -- 172.21.15.81:0/3232926907 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3990918263 0 0) 0x7f6f240c78b0 con 0x7f6f240cbd00 2024-05-27T07:00:58.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f23200640 1 --2- 172.21.15.81:0/3232926907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f24154200 0x7f6f24154610 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6f24073da0 tx=0x7f6f1c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0fb2ca830ac354 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:58.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f21400640 1 -- 172.21.15.81:0/3232926907 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c0b2070 con 0x7f6f24154200 2024-05-27T07:00:58.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f21400640 1 -- 172.21.15.81:0/3232926907 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f1c0a5900 con 0x7f6f24154200 2024-05-27T07:00:58.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f21400640 1 -- 172.21.15.81:0/3232926907 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c0a5cd0 con 0x7f6f24154200 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 -- 172.21.15.81:0/3232926907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f24154200 msgr2=0x7f6f24154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 --2- 172.21.15.81:0/3232926907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f24154200 0x7f6f24154610 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f6f24073da0 tx=0x7f6f1c01c930 comp rx=0 tx=0).stop 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 -- 172.21.15.81:0/3232926907 shutdown_connections 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 --2- 172.21.15.81:0/3232926907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f24154200 0x7f6f24154610 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 --2- 172.21.15.81:0/3232926907 >> v2:172.21.15.106:3300/0 conn(0x7f6f240ca770 0x7f6f240cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 -- 172.21.15.81:0/3232926907 >> 172.21.15.81:0/3232926907 conn(0x7f6f24068ae0 msgr2=0x7f6f24068f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:58.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 -- 172.21.15.81:0/3232926907 shutdown_connections 2024-05-27T07:00:58.190 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.199+0000 7f6f29400640 1 -- 172.21.15.81:0/3232926907 wait complete. 2024-05-27T07:00:58.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 Processor -- start 2024-05-27T07:00:58.191 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 -- start start 2024-05-27T07:00:58.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 0x7f6f241529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 0x7f6f2414ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f240d0640 con 0x7f6f240ca770 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6f240d04c0 con 0x7f6f240cbd00 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f6f24167590 con 0x7f6f24154200 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59238/0 (socket says 172.21.15.81:59238) 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 -- 172.21.15.81:0/278020452 learned_addr learned my addr 172.21.15.81:0/278020452 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f21e00640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 0x7f6f241529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 -- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 msgr2=0x7f6f2414ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f23200640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 0x7f6f2414ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 0x7f6f2414ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 -- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 msgr2=0x7f6f241529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 0x7f6f241529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 -- 172.21.15.81:0/278020452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2406f8e0 con 0x7f6f240ca770 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f23200640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 0x7f6f2414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f21e00640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 0x7f6f241529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f22800640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e870 tx=0x7f6f1400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14017070 con 0x7f6f240ca770 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f1400c940 con 0x7f6f240ca770 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.203+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f241f64c0 con 0x7f6f240ca770 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400ccf0 con 0x7f6f240ca770 2024-05-27T07:00:58.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2406f600 con 0x7f6f240ca770 2024-05-27T07:00:58.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f2406a560 con 0x7f6f240ca770 2024-05-27T07:00:58.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f6f1401a070 con 0x7f6f240ca770 2024-05-27T07:00:58.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f1aa00640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 0x7f6f04071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:00:58.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6f1408f7c0 con 0x7f6f240ca770 2024-05-27T07:00:58.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.207+0000 7f6f21e00640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 0x7f6f04071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:00:58.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.211+0000 7f6f21e00640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 0x7f6f04071940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6f2415c400 tx=0x7f6f10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:00:58.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.211+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6f1405c420 con 0x7f6f240ca770 2024-05-27T07:00:58.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.403+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f24164b50 con 0x7f6f240ca770 2024-05-27T07:00:58.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.403+0000 7f6f1aa00640 1 -- 172.21.15.81:0/278020452 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f6f1405bd20 con 0x7f6f240ca770 2024-05-27T07:00:58.394 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:00:58.394 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:00:58.394 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:00:58.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 msgr2=0x7f6f04071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 0x7f6f04071940 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6f2415c400 tx=0x7f6f10009040 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 msgr2=0x7f6f24162f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400e870 tx=0x7f6f1400ec70 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 shutdown_connections 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f6f24154200 0x7f6f2414ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f6f0406f430 0x7f6f04071940 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6f240cbd00 0x7f6f241529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 --2- 172.21.15.81:0/278020452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f240ca770 0x7f6f24162f00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 >> 172.21.15.81:0/278020452 conn(0x7f6f24068ae0 msgr2=0x7f6f24068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 shutdown_connections 2024-05-27T07:00:58.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:00:58.407+0000 7f6f29400640 1 -- 172.21.15.81:0/278020452 wait complete. 2024-05-27T07:00:58.415 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:00:59.728 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~295s 2024-05-27T07:01:00.416 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:01:00.576 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 Processor -- start 2024-05-27T07:01:00.577 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 -- start start 2024-05-27T07:01:00.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.578 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feae00c9f80 con 0x7feae006c790 2024-05-27T07:01:00.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7feae00c58b0 con 0x7feae00c8f90 2024-05-27T07:01:00.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.587+0000 7feae7a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7feae00c83f0 con 0x7feae015cb80 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3301/0 says I am v2:172.21.15.81:59750/0 (socket says 172.21.15.81:59750) 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 -- 172.21.15.81:0/890907243 learned_addr learned my addr 172.21.15.81:0/890907243 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.81:51948/0 (socket says 172.21.15.81:51948) 2024-05-27T07:01:00.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7fead7e00640 1 --1- 172.21.15.81:0/890907243 >> v1:172.21.15.106:6789/0 conn(0x7feae006c790 0x7feae00c8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.106:6789/0 says I am v1:172.21.15.81:34554/0 (socket says 172.21.15.81:34554) 2024-05-27T07:01:00.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 -- 172.21.15.81:0/890907243 >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 msgr2=0x7feae006cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:00.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 --2- 172.21.15.81:0/890907243 >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 -- 172.21.15.81:0/890907243 >> v1:172.21.15.106:6789/0 conn(0x7feae006c790 legacy=0x7feae00c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 -- 172.21.15.81:0/890907243 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae00c52f0 con 0x7feae015cb80 2024-05-27T07:01:00.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5a00640 1 --2- 172.21.15.81:0/890907243 >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:01:00.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.591+0000 7feae5000640 1 --2- 172.21.15.81:0/890907243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fead0009870 tx=0x7fead001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afa8f21b3abc8e2d server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:00.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feadfe00640 1 -- 172.21.15.81:0/890907243 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead00b0070 con 0x7feae015cb80 2024-05-27T07:01:00.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feadfe00640 1 -- 172.21.15.81:0/890907243 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead00ab600 con 0x7feae015cb80 2024-05-27T07:01:00.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feadfe00640 1 -- 172.21.15.81:0/890907243 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead00ab9d0 con 0x7feae015cb80 2024-05-27T07:01:00.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- 172.21.15.81:0/890907243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 msgr2=0x7feae0152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 --2- 172.21.15.81:0/890907243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fead0009870 tx=0x7fead001c560 comp rx=0 tx=0).stop 2024-05-27T07:01:00.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- 172.21.15.81:0/890907243 shutdown_connections 2024-05-27T07:01:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 --2- 172.21.15.81:0/890907243 >> v2:172.21.15.106:3300/0 conn(0x7feae00c8f90 0x7feae006cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 --2- 172.21.15.81:0/890907243 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae015cb80 0x7feae0152cb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- 172.21.15.81:0/890907243 >> 172.21.15.81:0/890907243 conn(0x7feae0070700 msgr2=0x7feae00cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- 172.21.15.81:0/890907243 shutdown_connections 2024-05-27T07:01:00.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- 172.21.15.81:0/890907243 wait complete. 2024-05-27T07:01:00.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 Processor -- start 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.595+0000 7feae7a00640 1 -- start start 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae00c8f90 0x7feae0151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 0x7feae01f9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feae00c9c80 con 0x7feae006c790 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feae00c9b00 con 0x7feae015cb80 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7feae01654f0 con 0x7feae00c8f90 2024-05-27T07:01:00.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59242/0 (socket says 172.21.15.81:59242) 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 -- 172.21.15.81:0/1368587984 learned_addr learned my addr 172.21.15.81:0/1368587984 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 0x7feae01f9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7fead7e00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae00c8f90 0x7feae0151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 -- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae00c8f90 msgr2=0x7feae0151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae00c8f90 0x7feae0151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 -- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 msgr2=0x7feae01f9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 0x7feae01f9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 -- 172.21.15.81:0/1368587984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae0151f60 con 0x7feae006c790 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 0x7feae01f9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-27T07:01:00.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae5000640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fead00ae040 tx=0x7fead001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead00b0040 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead00bc070 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae01f99a0 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead00b6660 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae01524f0 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.599+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae006a560 con 0x7feae006c790 2024-05-27T07:01:00.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.603+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fead00b6800 con 0x7feae006c790 2024-05-27T07:01:00.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.603+0000 7feadd600640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 0x7feabc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:00.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.603+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fead0129780 con 0x7feae006c790 2024-05-27T07:01:00.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.603+0000 7fead7e00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 0x7feabc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:00.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.603+0000 7fead7e00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 0x7feabc071910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feae00c7740 tx=0x7feac8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:00.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.607+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fead0016150 con 0x7feae006c790 2024-05-27T07:01:00.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feae015c960 con 0x7feae006c790 2024-05-27T07:01:00.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feadd600640 1 -- 172.21.15.81:0/1368587984 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fead00f63e0 con 0x7feae006c790 2024-05-27T07:01:00.792 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:01:00.792 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:01:00.793 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:01:00.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 msgr2=0x7feabc071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 0x7feabc071910 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feae00c7740 tx=0x7feac8009040 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 msgr2=0x7feae01f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fead00ae040 tx=0x7fead001ada0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 shutdown_connections 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7feae00c8f90 0x7feae0151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.803+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7feabc06f400 0x7feabc071910 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.807+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feae015cb80 0x7feae01f9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.807+0000 7feae7a00640 1 --2- 172.21.15.81:0/1368587984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feae006c790 0x7feae01f6850 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.807+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 >> 172.21.15.81:0/1368587984 conn(0x7feae0070700 msgr2=0x7feae00cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.807+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 shutdown_connections 2024-05-27T07:01:00.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:00.807+0000 7feae7a00640 1 -- 172.21.15.81:0/1368587984 wait complete. 2024-05-27T07:01:00.815 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:01:02.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 Processor -- start 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 -- start start 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 0x7faa000c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa000c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000c9c40 con 0x7faa000c9f30 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7faa000718c0 con 0x7faa000c9170 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7faa00071ba0 con 0x7faa000cad40 2024-05-27T07:01:02.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa04c00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7faa000c9f30 0x7faa000ca630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35356/0 (socket says 172.21.15.81:35356) 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa04c00640 1 -- 172.21.15.81:0/643527682 learned_addr learned my addr 172.21.15.81:0/643527682 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06000640 1 --2- 172.21.15.81:0/643527682 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa000c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa05600640 1 --2- 172.21.15.81:0/643527682 >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 0x7faa000c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7fa9ffe00640 1 -- 172.21.15.81:0/643527682 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566272989 0 0) 0x7faa000c9c40 con 0x7faa000c9f30 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7fa9ffe00640 1 -- 172.21.15.81:0/643527682 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e8003530 con 0x7faa000c9f30 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06000640 1 -- 172.21.15.81:0/643527682 >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 msgr2=0x7faa000c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06000640 1 --2- 172.21.15.81:0/643527682 >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 0x7faa000c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06000640 1 -- 172.21.15.81:0/643527682 >> v1:172.21.15.81:6789/0 conn(0x7faa000c9f30 legacy=0x7faa000ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa06000640 1 -- 172.21.15.81:0/643527682 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa00074b80 con 0x7faa000cad40 2024-05-27T07:01:02.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.987+0000 7faa05600640 1 --2- 172.21.15.81:0/643527682 >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 0x7faa000c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:01:02.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.991+0000 7faa06000640 1 --2- 172.21.15.81:0/643527682 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa000c8380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009d90 tx=0x7fa9f801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d66cefa9ba3abf6 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:02.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.991+0000 7fa9ffe00640 1 -- 172.21.15.81:0/643527682 <== mon.2 v2:172.21.15.106:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80b2070 con 0x7faa000cad40 2024-05-27T07:01:02.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.991+0000 7fa9ffe00640 1 -- 172.21.15.81:0/643527682 <== mon.2 v2:172.21.15.106:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f80a5940 con 0x7faa000cad40 2024-05-27T07:01:02.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.991+0000 7fa9ffe00640 1 -- 172.21.15.81:0/643527682 <== mon.2 v2:172.21.15.106:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80a5d10 con 0x7faa000cad40 2024-05-27T07:01:02.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- 172.21.15.81:0/643527682 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 msgr2=0x7faa000c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 --2- 172.21.15.81:0/643527682 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa000c8380 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009d90 tx=0x7fa9f801c970 comp rx=0 tx=0).stop 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- 172.21.15.81:0/643527682 shutdown_connections 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 --2- 172.21.15.81:0/643527682 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa000c8380 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 --2- 172.21.15.81:0/643527682 >> v2:172.21.15.106:3300/0 conn(0x7faa000c9170 0x7faa000c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- 172.21.15.81:0/643527682 >> 172.21.15.81:0/643527682 conn(0x7faa0006dea0 msgr2=0x7faa000ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- 172.21.15.81:0/643527682 shutdown_connections 2024-05-27T07:01:02.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- 172.21.15.81:0/643527682 wait complete. 2024-05-27T07:01:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 Processor -- start 2024-05-27T07:01:02.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.995+0000 7faa06a00640 1 -- start start 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 0x7faa00151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa0015c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faa00167980 con 0x7faa000c9170 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faa00167800 con 0x7faa000c9f30 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7faa00167b00 con 0x7faa000cad40 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59252/0 (socket says 172.21.15.81:59252) 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 -- 172.21.15.81:0/3695832837 learned_addr learned my addr 172.21.15.81:0/3695832837 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06000640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa0015c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa04c00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 0x7faa00151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 -- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 msgr2=0x7faa0015c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:02.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa0015c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 -- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 msgr2=0x7faa00151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 0x7faa00151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 -- 172.21.15.81:0/3695832837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000c6f60 con 0x7faa000c9170 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa04c00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 0x7faa00151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa05600640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e980 tx=0x7fa9f400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4017070 con 0x7faa000c9170 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f400ca00 con 0x7faa000c9170 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400cdd0 con 0x7faa000c9170 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0015b5d0 con 0x7faa000c9170 2024-05-27T07:01:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001f6750 con 0x7faa000c9170 2024-05-27T07:01:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa00050870 con 0x7faa000c9170 2024-05-27T07:01:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:02.999+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fa9f401a070 con 0x7faa000c9170 2024-05-27T07:01:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.003+0000 7fa9fd600640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 0x7fa9e0071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.003+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fa9f408f7d0 con 0x7faa000c9170 2024-05-27T07:01:02.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.003+0000 7faa04c00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 0x7fa9e0071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:02.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.003+0000 7faa04c00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 0x7fa9e0071910 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faa000c5270 tx=0x7fa9ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:02.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.007+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fa9f405bd50 con 0x7faa000c9170 2024-05-27T07:01:03.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.195+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa000733c0 con 0x7faa000c9170 2024-05-27T07:01:03.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.195+0000 7fa9fd600640 1 -- 172.21.15.81:0/3695832837 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fa9f4060050 con 0x7faa000c9170 2024-05-27T07:01:03.185 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:01:03.185 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:01:03.185 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 msgr2=0x7fa9e0071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 0x7fa9e0071910 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faa000c5270 tx=0x7fa9ec009040 comp rx=0 tx=0).stop 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 msgr2=0x7faa000c6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e980 tx=0x7fa9f400ee40 comp rx=0 tx=0).stop 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 shutdown_connections 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7faa000cad40 0x7faa0015c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fa9e006f400 0x7fa9e0071910 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:03.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faa000c9f30 0x7faa00151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:03.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 --2- 172.21.15.81:0/3695832837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa000c9170 0x7faa000c6a20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:03.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 >> 172.21.15.81:0/3695832837 conn(0x7faa0006dea0 msgr2=0x7faa000cbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:03.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 shutdown_connections 2024-05-27T07:01:03.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:03.199+0000 7faa06a00640 1 -- 172.21.15.81:0/3695832837 wait complete. 2024-05-27T07:01:03.207 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:01:05.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:01:05.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:01:05.261 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:01:05.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 Processor -- start 2024-05-27T07:01:05.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 -- start start 2024-05-27T07:01:05.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30072430 0x7f1e30159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f1e30161cd0 0x7f1e301640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e30072840 con 0x7f1e3015a040 2024-05-27T07:01:05.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f1e300c5040 con 0x7f1e30161cd0 2024-05-27T07:01:05.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.379+0000 7f1e35c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e300c75c0 con 0x7f1e30072430 2024-05-27T07:01:05.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1e3015a040 0x7f1e300c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43088/0 (socket says 172.21.15.81:43088) 2024-05-27T07:01:05.371 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 -- 172.21.15.81:0/2181205137 learned_addr learned my addr 172.21.15.81:0/2181205137 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:01:05.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30072430 0x7f1e30159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2f400640 1 --2- 172.21.15.81:0/2181205137 >> v2:172.21.15.106:3300/0 conn(0x7f1e30161cd0 0x7f1e301640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058893520 0 0) 0x7f1e30072840 con 0x7f1e3015a040 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e14003530 con 0x7f1e3015a040 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 954287517 0 0) 0x7f1e14003530 con 0x7f1e3015a040 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30072430 msgr2=0x7f1e30159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30072430 0x7f1e30159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 >> v2:172.21.15.106:3300/0 conn(0x7f1e30161cd0 msgr2=0x7f1e301640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 --2- 172.21.15.81:0/2181205137 >> v2:172.21.15.106:3300/0 conn(0x7f1e30161cd0 0x7f1e301640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300c5270 con 0x7f1e3015a040 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2072710855 0 0) 0x7f1e1c001700 con 0x7f1e3015a040 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 >> v1:172.21.15.81:6789/0 conn(0x7f1e3015a040 legacy=0x7f1e300c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.373 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e14003a10 0x7f1e14003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e14004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e14005160 0x7f1e1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300c75c0 con 0x7f1e14004810 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1400bf90 con 0x7f1e14003a10 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e1400c190 con 0x7f1e14005160 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30072430 0x7f1e30159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2f400640 1 --2- 172.21.15.81:0/2181205137 >> v2:172.21.15.106:3300/0 conn(0x7f1e30161cd0 0x7f1e301640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e14004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e14003a10 0x7f1e14003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2f400640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e14005160 0x7f1e1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 -- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e14005160 msgr2=0x7f1e1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e14005160 0x7f1e1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.374 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 -- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e14003a10 msgr2=0x7f1e14003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e14003a10 0x7f1e14003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.383+0000 7f1e2e000640 1 -- 172.21.15.81:0/2181205137 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e14003530 con 0x7f1e14004810 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e2e000640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e14004c20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1e3006f550 tx=0x7f1e1c01f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b4a45e8a597aa server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c007d60 con 0x7f1e14004810 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e1c005ae0 con 0x7f1e14004810 2024-05-27T07:01:05.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e2d600640 1 -- 172.21.15.81:0/2181205137 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c007070 con 0x7f1e14004810 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 -- 172.21.15.81:0/2181205137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 msgr2=0x7f1e14004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 --2- 172.21.15.81:0/2181205137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e14004c20 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1e3006f550 tx=0x7f1e1c01f2f0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e2f400640 1 -- 172.21.15.81:0/2181205137 reap_dead start 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 -- 172.21.15.81:0/2181205137 shutdown_connections 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 -- 172.21.15.81:0/2181205137 >> 172.21.15.81:0/2181205137 conn(0x7f1e30070700 msgr2=0x7f1e300cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:05.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 -- 172.21.15.81:0/2181205137 shutdown_connections 2024-05-27T07:01:05.377 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.387+0000 7f1e35c00640 1 -- 172.21.15.81:0/2181205137 wait complete. 2024-05-27T07:01:05.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 Processor -- start 2024-05-27T07:01:05.378 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- start start 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e301f2af0 0x7f1e300cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30158600 0x7f1e30149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300c7af0 con 0x7f1e14004810 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e300c7dd0 con 0x7f1e301f2af0 2024-05-27T07:01:05.381 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f1e301595e0 con 0x7f1e30158600 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38444/0 (socket says 172.21.15.81:38444) 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 -- 172.21.15.81:0/10067517 learned_addr learned my addr 172.21.15.81:0/10067517 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2e000640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e301f2af0 0x7f1e300cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2f400640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30158600 0x7f1e30149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 -- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30158600 msgr2=0x7f1e30149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30158600 0x7f1e30149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 -- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e301f2af0 msgr2=0x7f1e300cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e301f2af0 0x7f1e300cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 -- 172.21.15.81:0/10067517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3014df20 con 0x7f1e14004810 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e2ea00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f1e20011630 tx=0x7f1e20011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:05.382 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000f920 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e2000fac0 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000fe70 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e300cc830 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.391+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e3014e4c0 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e3006a560 con 0x7f1e14004810 2024-05-27T07:01:05.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f1e20015460 con 0x7f1e14004810 2024-05-27T07:01:05.384 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e26a00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 0x7f1e10071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:05.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f1e2008fb00 con 0x7f1e14004810 2024-05-27T07:01:05.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e2e000640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 0x7f1e10071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:05.385 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.395+0000 7f1e2e000640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 0x7f1e10071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0af000 tx=0x7f1e1c0b2040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:05.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.399+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f1e20019030 con 0x7f1e14004810 2024-05-27T07:01:05.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e30153bc0 con 0x7f1e14004810 2024-05-27T07:01:05.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e26a00640 1 -- 172.21.15.81:0/10067517 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7f1e2005c110 con 0x7f1e14004810 2024-05-27T07:01:05.580 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:01:05.581 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:01:05.581 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:01:05.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 msgr2=0x7f1e10071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 0x7f1e10071910 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c0af000 tx=0x7f1e1c0b2040 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 msgr2=0x7f1e30151a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f1e20011630 tx=0x7f1e20011af0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.591+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 shutdown_connections 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f1e30158600 0x7f1e30149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f1e1006f400 0x7f1e10071910 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e301f2af0 0x7f1e300cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 --2- 172.21.15.81:0/10067517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e14004810 0x7f1e30151a50 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 >> 172.21.15.81:0/10067517 conn(0x7f1e30070700 msgr2=0x7f1e301532d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 shutdown_connections 2024-05-27T07:01:05.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:05.595+0000 7f1e35c00640 1 -- 172.21.15.81:0/10067517 wait complete. 2024-05-27T07:01:05.603 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 49, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-05-27T06:45:02.016832+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 49, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4870": { "addr": "172.21.15.106:6833/808799882", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6832", "nonce": 808799882, "type": "v2" }, { "addr": "172.21.15.106:6833", "nonce": 808799882, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4870, "incarnation": 24, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:active", "state_seq": 14 }, "gid_4949": { "addr": "172.21.15.106:6837/3619396855", "addrs": { "addrvec": [ { "addr": "172.21.15.106:6835", "nonce": 3619396855, "type": "v2" }, { "addr": "172.21.15.106:6837", "nonce": 3619396855, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4949, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 1, "state": "up:active", "state_seq": 14 }, "gid_5049": { "addr": "172.21.15.81:6838/3566669304", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3566669304, "type": "v2" }, { "addr": "172.21.15.81:6838", "nonce": 3566669304, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5049, "incarnation": 30, "join_fscid": -1, "name": "e", "rank": 3, "state": "up:active", "state_seq": 41 }, "gid_5184": { "addr": "172.21.15.81:6835/2124140394", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2124140394, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2124140394, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5184, "incarnation": 36, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 35 }, "gid_5448": { "addr": "172.21.15.81:6839/1991504158", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6837", "nonce": 1991504158, "type": "v2" }, { "addr": "172.21.15.81:6839", "nonce": 1991504158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5448, "incarnation": 48, "join_fscid": -1, "laggy_since": "2024-05-27T06:56:18.384986+0000", "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-05-27T06:56:18.384986+0000", "qdb_cluster": [ 5184, 5049, 4949, 4870 ], "qdb_leader": 4949, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5448, "mds_1": 4949, "mds_2": 5184, "mds_3": 5049, "mds_4": 4870 } } } ], "standbys": [] } 2024-05-27T07:01:06.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~302s 2024-05-27T07:01:06.441 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-05-27T07:01:06.441 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T07:01:06.441 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:01:06.451 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-27T07:01:06.579 INFO:tasks.workunit.client.0.smithi081.stderr:fatal: Unable to create temporary file '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/objects/pack/tmp_idx_XXXXXX': Transport endpoint is not connected 2024-05-27T07:01:06.582 INFO:tasks.workunit.client.0.smithi081.stderr:fatal: fetch-pack: invalid index-pack output 2024-05-27T07:01:06.582 INFO:tasks.workunit.client.0.smithi081.stderr:warning: unable to unlink '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph/.git/shallow_NwRwhb': Transport endpoint is not connected 2024-05-27T07:01:06.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.659+0000 7f6978c00640 1 Processor -- start 2024-05-27T07:01:06.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.659+0000 7f6978c00640 1 -- start start 2024-05-27T07:01:06.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6978c00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f6974154660 0x7f697415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:06.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6978c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697415a620 con 0x7f69740c9ab0 2024-05-27T07:01:06.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6978c00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69740c7bf0 con 0x7f69740c96a0 2024-05-27T07:01:06.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6978c00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f69740c9ec0 con 0x7f6974154660 2024-05-27T07:01:06.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6972800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f69740c9ab0 0x7f6974164fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43092/0 (socket says 172.21.15.81:43092) 2024-05-27T07:01:06.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6972800640 1 -- 172.21.15.81:0/865043317 learned_addr learned my addr 172.21.15.81:0/865043317 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:01:06.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971e00640 1 --2- 172.21.15.81:0/865043317 >> v2:172.21.15.106:3300/0 conn(0x7f6974154660 0x7f697415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090787298 0 0) 0x7f697415a620 con 0x7f69740c9ab0 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6958003530 con 0x7f69740c9ab0 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2510462313 0 0) 0x7f6958003530 con 0x7f69740c9ab0 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971e00640 1 -- 172.21.15.81:0/865043317 >> v1:172.21.15.106:6789/0 conn(0x7f69740c96a0 legacy=0x7f69740cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971e00640 1 -- 172.21.15.81:0/865043317 >> v1:172.21.15.81:6789/0 conn(0x7f69740c9ab0 legacy=0x7f6974164fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.652 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.663+0000 7f6971e00640 1 -- 172.21.15.81:0/865043317 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69740c7ed0 con 0x7f6974154660 2024-05-27T07:01:06.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6971e00640 1 --2- 172.21.15.81:0/865043317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f697415bb10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69640097c0 tx=0x7f696401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c032932373433c server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:06.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69640b0070 con 0x7f6974154660 2024-05-27T07:01:06.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69640ab400 con 0x7f6974154660 2024-05-27T07:01:06.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6971400640 1 -- 172.21.15.81:0/865043317 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696c001aa0 con 0x7f6974154660 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- 172.21.15.81:0/865043317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 msgr2=0x7f697415bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 --2- 172.21.15.81:0/865043317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f697415bb10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f69640097c0 tx=0x7f696401c3c0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- 172.21.15.81:0/865043317 shutdown_connections 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 --2- 172.21.15.81:0/865043317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f697415bb10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- 172.21.15.81:0/865043317 >> 172.21.15.81:0/865043317 conn(0x7f6974068ab0 msgr2=0x7f6974068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- 172.21.15.81:0/865043317 shutdown_connections 2024-05-27T07:01:06.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- 172.21.15.81:0/865043317 wait complete. 2024-05-27T07:01:06.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 Processor -- start 2024-05-27T07:01:06.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.667+0000 7f6978c00640 1 -- start start 2024-05-27T07:01:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 0x7f69741f9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:06.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f6974150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6974154a70 con 0x7f69740c96a0 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f69740c9210 con 0x7f6974154660 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f697415c050 con 0x7f69740c9ab0 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38452/0 (socket says 172.21.15.81:38452) 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 -- 172.21.15.81:0/3599119797 learned_addr learned my addr 172.21.15.81:0/3599119797 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f696be00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 0x7f69741f9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6972800640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f6974150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 -- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 msgr2=0x7f69741f9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 0x7f69741f9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 -- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 msgr2=0x7f6974150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f6974150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 -- 172.21.15.81:0/3599119797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69741f9b70 con 0x7f69740c96a0 2024-05-27T07:01:06.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f696be00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 0x7f69741f9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-27T07:01:06.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6971e00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6964000c10 tx=0x7f6964019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69640b0040 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69640036a0 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964003a50 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69740cc470 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69740cc730 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.671+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697406a560 con 0x7f69740c96a0 2024-05-27T07:01:06.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.675+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7f69640bd020 con 0x7f69740c96a0 2024-05-27T07:01:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.675+0000 7f696a000640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 0x7f6950071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:06.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.675+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7f6964129950 con 0x7f69740c96a0 2024-05-27T07:01:06.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.675+0000 7f696be00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 0x7f6950071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:06.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.675+0000 7f696be00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 0x7f6950071940 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f69740c6a40 tx=0x7f695c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:06.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.683+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7f6964015fb0 con 0x7f69740c96a0 2024-05-27T07:01:06.702 INFO:tasks.workunit.client.0.smithi081.stderr:+ retry 2024-05-27T07:01:06.702 INFO:tasks.workunit.client.0.smithi081.stderr:+ rm -rf ceph 2024-05-27T07:01:06.703 INFO:tasks.workunit.client.0.smithi081.stderr:rm: cannot remove 'ceph': Transport endpoint is not connected 2024-05-27T07:01:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:01:06.704 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2024-05-27T07:01:06.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-27T07:01:06.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.855+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f697415d9f0 con 0x7f69740c96a0 2024-05-27T07:01:06.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f696a000640 1 -- 172.21.15.81:0/3599119797 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v60) ==== 81+0+728 (secure 0 0 0) 0x7f69640f66a0 con 0x7f69740c96a0 2024-05-27T07:01:06.846 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6834/730205609 2024-05-28T06:55:53.380458+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6839/328699254 2024-05-28T06:53:58.363620+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6837/328699254 2024-05-28T06:53:58.363620+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6837/13170464 2024-05-28T06:53:03.351113+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6835/13170464 2024-05-28T06:53:03.351113+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6834/2990302599 2024-05-28T06:50:48.327565+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6838/1291517441 2024-05-28T06:48:13.298584+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6835/2990302599 2024-05-28T06:50:48.327565+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6836/1291517441 2024-05-28T06:48:13.298584+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6836/730205609 2024-05-28T06:55:53.380458+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6833/353223641 2024-05-28T06:47:28.290611+0000 2024-05-27T07:01:06.847 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6832/353223641 2024-05-28T06:47:28.290611+0000 2024-05-27T07:01:06.848 INFO:teuthology.orchestra.run.smithi081.stderr:listed 12 entries 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 msgr2=0x7f6950071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 0x7f6950071940 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f69740c6a40 tx=0x7f695c009040 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 msgr2=0x7f69740cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6964000c10 tx=0x7f6964019160 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 shutdown_connections 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f69740c9ab0 0x7f69741f9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7f695006f430 0x7f6950071940 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6974154660 0x7f6974150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 --2- 172.21.15.81:0/3599119797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69740c96a0 0x7f69740cb2f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 >> 172.21.15.81:0/3599119797 conn(0x7f6974068ab0 msgr2=0x7f69740cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 shutdown_connections 2024-05-27T07:01:06.849 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:06.859+0000 7f6978c00640 1 -- 172.21.15.81:0/3599119797 wait complete. 2024-05-27T07:01:06.869 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T07:01:06.869 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:01:06.878 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-27T07:01:07.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 Processor -- start 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 -- start start 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7fb3d80ca930 0x7fb3d80d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d8156be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d80ca310 con 0x7fb3d815c910 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d80cb090 con 0x7fb3d80ca930 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3df200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d80ca5b0 con 0x7fb3d81567d0 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3dd200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb3d815c910 0x7fb3d815cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43106/0 (socket says 172.21.15.81:43106) 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3dd200640 1 -- 172.21.15.81:0/275488164 learned_addr learned my addr 172.21.15.81:0/275488164 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7400640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d8156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/275488164 >> v2:172.21.15.106:3300/0 conn(0x7fb3d80ca930 0x7fb3d80d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d6a00640 1 -- 172.21.15.81:0/275488164 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963853982 0 0) 0x7fb3d80ca310 con 0x7fb3d815c910 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d6a00640 1 -- 172.21.15.81:0/275488164 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3bc003530 con 0x7fb3d815c910 2024-05-27T07:01:07.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7e00640 1 -- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 msgr2=0x7fb3d8156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:07.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d8156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7e00640 1 -- 172.21.15.81:0/275488164 >> v1:172.21.15.81:6789/0 conn(0x7fb3d815c910 legacy=0x7fb3d815cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.083+0000 7fb3d7e00640 1 -- 172.21.15.81:0/275488164 --> v2:172.21.15.106:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d80cb350 con 0x7fb3d80ca930 2024-05-27T07:01:07.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3d7400640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d8156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-27T07:01:07.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80d00c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80097c0 tx=0x7fb3c801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18b48554846922eb server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3d6a00640 1 -- 172.21.15.81:0/275488164 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80b0070 con 0x7fb3d80ca930 2024-05-27T07:01:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3d6a00640 1 -- 172.21.15.81:0/275488164 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c80ab600 con 0x7fb3d80ca930 2024-05-27T07:01:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3d6a00640 1 -- 172.21.15.81:0/275488164 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c80ab9d0 con 0x7fb3d80ca930 2024-05-27T07:01:07.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3df200640 1 -- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 msgr2=0x7fb3d80d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.087+0000 7fb3df200640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80d00c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80097c0 tx=0x7fb3c801c4a0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- 172.21.15.81:0/275488164 shutdown_connections 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d8156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 --2- 172.21.15.81:0/275488164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80d00c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- 172.21.15.81:0/275488164 >> 172.21.15.81:0/275488164 conn(0x7fb3d8068ab0 msgr2=0x7fb3d8068ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- 172.21.15.81:0/275488164 shutdown_connections 2024-05-27T07:01:07.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- 172.21.15.81:0/275488164 wait complete. 2024-05-27T07:01:07.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 Processor -- start 2024-05-27T07:01:07.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- start start 2024-05-27T07:01:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d80c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d80d0600 con 0x7fb3d815c910 2024-05-27T07:01:07.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d80c9d10 con 0x7fb3d80ca930 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.091+0000 7fb3df200640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d80d0780 con 0x7fb3d81567d0 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38454/0 (socket says 172.21.15.81:38454) 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 -- 172.21.15.81:0/305182223 learned_addr learned my addr 172.21.15.81:0/305182223 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3d7400640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d80c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 -- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 msgr2=0x7fb3d80c38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d80c38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 -- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 msgr2=0x7fb3d80c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 -- 172.21.15.81:0/305182223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d81f91f0 con 0x7fb3d815c910 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3dd200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000cb00 tx=0x7fb3d000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d000ac40 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d81f94d0 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d000ade0 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0012600 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d8153650 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d806a560 con 0x7fb3d815c910 2024-05-27T07:01:07.085 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.095+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fb3d00127a0 con 0x7fb3d815c910 2024-05-27T07:01:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.099+0000 7fb3cfe00640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 0x7fb3b8071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.099+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fb3d008f940 con 0x7fb3d815c910 2024-05-27T07:01:07.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.099+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 0x7fb3b8071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.099+0000 7fb3d7e00640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 0x7fb3b8071910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ad000 tx=0x7fb3c80a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.092 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.103+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fb3d005c000 con 0x7fb3d815c910 2024-05-27T07:01:07.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.271+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb3d80726f0 con 0x7fb3d815c910 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3cfe00640 1 -- 172.21.15.81:0/305182223 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v60) ==== 81+0+728 (secure 0 0 0) 0x7fb3d0060300 con 0x7fb3d815c910 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6834/730205609 2024-05-28T06:55:53.380458+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6839/328699254 2024-05-28T06:53:58.363620+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6837/328699254 2024-05-28T06:53:58.363620+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6837/13170464 2024-05-28T06:53:03.351113+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6835/13170464 2024-05-28T06:53:03.351113+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6834/2990302599 2024-05-28T06:50:48.327565+0000 2024-05-27T07:01:07.263 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6838/1291517441 2024-05-28T06:48:13.298584+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6835/2990302599 2024-05-28T06:50:48.327565+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6836/1291517441 2024-05-28T06:48:13.298584+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6836/730205609 2024-05-28T06:55:53.380458+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6833/353223641 2024-05-28T06:47:28.290611+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.106:6832/353223641 2024-05-28T06:47:28.290611+0000 2024-05-27T07:01:07.264 INFO:teuthology.orchestra.run.smithi081.stderr:listed 12 entries 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 msgr2=0x7fb3b8071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 0x7fb3b8071910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb3c80ad000 tx=0x7fb3c80a6000 comp rx=0 tx=0).stop 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 msgr2=0x7fb3d80c4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000cb00 tx=0x7fb3d000a060 comp rx=0 tx=0).stop 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 shutdown_connections 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fb3d81567d0 0x7fb3d80c38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fb3b806f400 0x7fb3b8071910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3d815c910 0x7fb3d80c4b20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 --2- 172.21.15.81:0/305182223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb3d80ca930 0x7fb3d80c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 >> 172.21.15.81:0/305182223 conn(0x7fb3d8068ab0 msgr2=0x7fb3d806dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:07.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 shutdown_connections 2024-05-27T07:01:07.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.275+0000 7fb3df200640 1 -- 172.21.15.81:0/305182223 wait complete. 2024-05-27T07:01:07.285 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi081.front.sepia.ceph.com... 2024-05-27T07:01:07.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:01:07.286 DEBUG:teuthology.orchestra.run.smithi081:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-27T07:01:07.322 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:01:07.322 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:01:07.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:01:07.323 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-27T07:01:07.603 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:01:07.606 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-05-27T07:01:07.819 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cad2714540fc496eb2736d3d148b6084 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-05-27T07:01:07.821 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-05-27T07:01:07.831 INFO:tasks.mds_thrash:joining mds_thrasher 2024-05-27T07:01:07.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.763+0000 7fd7f8a00640 1 Processor -- start 2024-05-27T07:01:07.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.763+0000 7fd7f8a00640 1 -- start start 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f8a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd7f40744a0 0x7fd7f40c9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f8a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f40ca1e0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f8a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f40ca480 con 0x7fd7f4071f90 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f8a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f415adc0 con 0x7fd7f40744a0 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f3400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd7f40744a0 0x7fd7f40c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f2a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd7f40c9b70 0x7fd7f4154490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43122/0 (socket says 172.21.15.81:43122) 2024-05-27T07:01:07.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f2a00640 1 -- 172.21.15.81:0/316774635 learned_addr learned my addr 172.21.15.81:0/316774635 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f3400640 1 -- 172.21.15.81:0/316774635 >> v1:172.21.15.106:6789/0 conn(0x7fd7f4071f90 legacy=0x7fd7f4164ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f3400640 1 -- 172.21.15.81:0/316774635 >> v1:172.21.15.81:6789/0 conn(0x7fd7f40c9b70 legacy=0x7fd7f4154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f2000640 1 -- 172.21.15.81:0/316774635 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110925222 0 0) 0x7fd7f40ca1e0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f3400640 1 -- 172.21.15.81:0/316774635 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f40ca740 con 0x7fd7f40744a0 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.767+0000 7fd7f3400640 1 --2- 172.21.15.81:0/316774635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40744a0 0x7fd7f40c9630 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40097a0 tx=0x7fd7e401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6801a342e72e702 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f2000640 1 -- 172.21.15.81:0/316774635 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40b0070 con 0x7fd7f40744a0 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f2000640 1 -- 172.21.15.81:0/316774635 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40ab600 con 0x7fd7f40744a0 2024-05-27T07:01:07.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f2000640 1 -- 172.21.15.81:0/316774635 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e40ab9d0 con 0x7fd7f40744a0 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- 172.21.15.81:0/316774635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40744a0 msgr2=0x7fd7f40c9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/316774635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40744a0 0x7fd7f40c9630 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40097a0 tx=0x7fd7e401c480 comp rx=0 tx=0).stop 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- 172.21.15.81:0/316774635 shutdown_connections 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/316774635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40744a0 0x7fd7f40c9630 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- 172.21.15.81:0/316774635 >> 172.21.15.81:0/316774635 conn(0x7fd7f4068cb0 msgr2=0x7fd7f40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- 172.21.15.81:0/316774635 shutdown_connections 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- 172.21.15.81:0/316774635 wait complete. 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 Processor -- start 2024-05-27T07:01:07.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.771+0000 7fd7f8a00640 1 -- start start 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4071f90 0x7fd7f4161780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f40744a0 0x7fd7f414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f4154ba0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f40727b0 con 0x7fd7f40744a0 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f40748b0 con 0x7fd7f4071f90 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38472/0 (socket says 172.21.15.81:38472) 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 -- 172.21.15.81:0/2092926455 learned_addr learned my addr 172.21.15.81:0/2092926455 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3400640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4071f90 0x7fd7f4161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f2a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f40744a0 0x7fd7f414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 -- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4071f90 msgr2=0x7fd7f4161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4071f90 0x7fd7f4161780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 -- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f40744a0 msgr2=0x7fd7f414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f40744a0 0x7fd7f414ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 -- 172.21.15.81:0/2092926455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f4161cc0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f3e00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00ba10 tx=0x7fd7ec00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec019070 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ec00cb40 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f41606e0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec014440 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f4160ca0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.775+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f414f9d0 con 0x7fd7f40c9b70 2024-05-27T07:01:07.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.779+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 93207+0+0 (secure 0 0 0) 0x7fd7ec01c070 con 0x7fd7f40c9b70 2024-05-27T07:01:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.779+0000 7fd7eb400640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.779+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6341+0+0 (secure 0 0 0) 0x7fd7ec08f400 con 0x7fd7f40c9b70 2024-05-27T07:01:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.779+0000 7fd7f3400640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-27T07:01:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.779+0000 7fd7f3400640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 0x7fd7d4071910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40ad000 tx=0x7fd7e40a6000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-27T07:01:07.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.783+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180029 (secure 0 0 0) 0x7fd7ec05c150 con 0x7fd7f40c9b70 2024-05-27T07:01:07.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7f4050870 con 0x7fd7f40c9b70 2024-05-27T07:01:07.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7eb400640 1 -- 172.21.15.81:0/2092926455 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6135 (secure 0 0 0) 0x7fd7ec05ba50 con 0x7fd7f40c9b70 2024-05-27T07:01:07.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-05-27T07:01:07.960 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-27T06:45:02.016832+0000","modified":"2024-05-27T06:56:18.384986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5448,"mds_1":4949,"mds_2":5184,"mds_3":5049,"mds_4":4870},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4870":{"gid":4870,"name":"d","rank":4,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6833/808799882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":808799882},{"type":"v1","addr":"172.21.15.106:6833","nonce":808799882}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4949":{"gid":4949,"name":"b","rank":1,"incarnation":42,"state":"up:active","state_seq":14,"addr":"172.21.15.106:6837/3619396855","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6835","nonce":3619396855},{"type":"v1","addr":"172.21.15.106:6837","nonce":3619396855}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5049":{"gid":5049,"name":"e","rank":3,"incarnation":30,"state":"up:active","state_seq":41,"addr":"172.21.15.81:6838/3566669304","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3566669304},{"type":"v1","addr":"172.21.15.81:6838","nonce":3566669304}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5184":{"gid":5184,"name":"c","rank":2,"incarnation":36,"state":"up:active","state_seq":35,"addr":"172.21.15.81:6835/2124140394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2124140394},{"type":"v1","addr":"172.21.15.81:6835","nonce":2124140394}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5448":{"gid":5448,"name":"a","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6839/1991504158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6837","nonce":1991504158},{"type":"v1","addr":"172.21.15.81:6839","nonce":1991504158}]},"join_fscid":-1,"laggy_since":"2024-05-27T06:56:18.384986+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4949,"qdb_cluster":[5184,5049,4949,4870]},"id":1}]} 2024-05-27T07:01:07.961 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 49 2024-05-27T07:01:07.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 msgr2=0x7fd7d4071910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 0x7fd7d4071910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40ad000 tx=0x7fd7e40a6000 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 msgr2=0x7fd7f414f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00ba10 tx=0x7fd7ec00bed0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 shutdown_connections 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:6832/1352206918,v1:172.21.15.81:6833/1352206918] conn(0x7fd7d406f400 0x7fd7d4071910 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7f40c9b70 0x7fd7f414f490 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7f40744a0 0x7fd7f414ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 --2- 172.21.15.81:0/2092926455 >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7fd7f4071f90 0x7fd7f4161780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:07.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.971+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 >> 172.21.15.81:0/2092926455 conn(0x7fd7f4068cb0 msgr2=0x7fd7f4159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-27T07:01:07.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.975+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 shutdown_connections 2024-05-27T07:01:07.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:07.975+0000 7fd7f8a00640 1 -- 172.21.15.81:0/2092926455 wait complete. 2024-05-27T07:01:07.982 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-05-27T07:01:09.244 INFO:tasks.ceph.osd.0:Sent signal 15 2024-05-27T07:01:09.244 INFO:tasks.ceph.osd.1:Sent signal 15 2024-05-27T07:01:09.245 INFO:tasks.ceph.osd.2:Sent signal 15 2024-05-27T07:01:09.245 INFO:tasks.ceph.osd.3:Sent signal 15 2024-05-27T07:01:09.245 INFO:tasks.ceph.osd.4:Sent signal 15 2024-05-27T07:01:09.245 INFO:tasks.ceph.osd.5:Sent signal 15 2024-05-27T07:01:09.246 INFO:tasks.ceph.osd.6:Sent signal 15 2024-05-27T07:01:09.246 INFO:tasks.ceph.osd.7:Sent signal 15 2024-05-27T07:01:09.246 INFO:tasks.ceph.mds.c:Sent signal 15 2024-05-27T07:01:09.246 INFO:tasks.ceph.mds.e:Sent signal 15 2024-05-27T07:01:09.247 INFO:tasks.ceph.mds.b:Sent signal 15 2024-05-27T07:01:09.247 INFO:tasks.ceph.mds.d:Sent signal 15 2024-05-27T07:01:09.247 INFO:tasks.ceph.mon.a:Sent signal 15 2024-05-27T07:01:09.248 INFO:tasks.ceph.mon.b:Sent signal 15 2024-05-27T07:01:09.248 INFO:tasks.ceph.mon.c:Sent signal 15 2024-05-27T07:01:09.248 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-05-27T07:01:09.248 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-05-27T07:01:09.249 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fe643000640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 28253) UID: 0 2024-05-27T07:01:09.249 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fe643000640 -1 osd.4 60 *** Got signal Terminated *** 2024-05-27T07:01:09.249 INFO:tasks.ceph.osd.4.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fe643000640 -1 osd.4 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.250 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1aec200640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 28256) UID: 0 2024-05-27T07:01:09.250 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1aec200640 -1 osd.5 60 *** Got signal Terminated *** 2024-05-27T07:01:09.250 INFO:tasks.ceph.osd.5.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1aec200640 -1 osd.5 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.250 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1333e00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 28255) UID: 0 2024-05-27T07:01:09.251 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1333e00640 -1 osd.6 60 *** Got signal Terminated *** 2024-05-27T07:01:09.251 INFO:tasks.ceph.osd.6.smithi106.stderr:2024-05-27T07:01:09.257+0000 7f1333e00640 -1 osd.6 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.251 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fce04600640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 28259) UID: 0 2024-05-27T07:01:09.251 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fce04600640 -1 osd.7 60 *** Got signal Terminated *** 2024-05-27T07:01:09.251 INFO:tasks.ceph.osd.7.smithi106.stderr:2024-05-27T07:01:09.257+0000 7fce04600640 -1 osd.7 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.252 INFO:tasks.ceph.mds.b.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fbd9ea00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i b (PID: 32152) UID: 0 2024-05-27T07:01:09.252 INFO:tasks.ceph.mds.b.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fbd9ea00640 -1 mds.b *** got signal Terminated *** 2024-05-27T07:01:09.252 INFO:tasks.ceph.mds.d.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fbd97200640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i d (PID: 32066) UID: 0 2024-05-27T07:01:09.252 INFO:tasks.ceph.mds.d.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fbd97200640 -1 mds.d *** got signal Terminated *** 2024-05-27T07:01:09.253 INFO:tasks.ceph.mon.b.smithi106.stderr:2024-05-27T07:01:09.261+0000 7f66a5000640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 27933) UID: 0 2024-05-27T07:01:09.253 INFO:tasks.ceph.mon.b.smithi106.stderr:2024-05-27T07:01:09.261+0000 7f66a5000640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-05-27T07:01:09.253 INFO:tasks.ceph.mon.c.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fcd98a00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 27935) UID: 0 2024-05-27T07:01:09.253 INFO:tasks.ceph.mon.c.smithi106.stderr:2024-05-27T07:01:09.261+0000 7fcd98a00640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-05-27T07:01:09.254 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T07:01:09.255+0000 7fc087a00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 28035) UID: 0 2024-05-27T07:01:09.254 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T07:01:09.255+0000 7fc087a00640 -1 osd.0 60 *** Got signal Terminated *** 2024-05-27T07:01:09.254 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-05-27T07:01:09.255+0000 7fc087a00640 -1 osd.0 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.254 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T07:01:09.255+0000 7efcdd200640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 28037) UID: 0 2024-05-27T07:01:09.255 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T07:01:09.255+0000 7efcdd200640 -1 osd.1 60 *** Got signal Terminated *** 2024-05-27T07:01:09.255 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-05-27T07:01:09.255+0000 7efcdd200640 -1 osd.1 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.255 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T07:01:09.255+0000 7f5d9c000640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 28040) UID: 0 2024-05-27T07:01:09.255 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T07:01:09.255+0000 7f5d9c000640 -1 osd.2 60 *** Got signal Terminated *** 2024-05-27T07:01:09.256 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-05-27T07:01:09.255+0000 7f5d9c000640 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.256 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T07:01:09.259+0000 7fdfec600640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 28039) UID: 0 2024-05-27T07:01:09.256 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T07:01:09.259+0000 7fdfec600640 -1 osd.3 60 *** Got signal Terminated *** 2024-05-27T07:01:09.256 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-05-27T07:01:09.259+0000 7fdfec600640 -1 osd.3 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-27T07:01:09.257 INFO:tasks.ceph.mds.c.smithi081.stderr:2024-05-27T07:01:09.259+0000 7f929d400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 37612) UID: 0 2024-05-27T07:01:09.257 INFO:tasks.ceph.mds.c.smithi081.stderr:2024-05-27T07:01:09.259+0000 7f929d400640 -1 mds.c *** got signal Terminated *** 2024-05-27T07:01:09.257 INFO:tasks.ceph.mds.e.smithi081.stderr:2024-05-27T07:01:09.259+0000 7ff5f0a00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i e (PID: 35737) UID: 0 2024-05-27T07:01:09.257 INFO:tasks.ceph.mds.e.smithi081.stderr:2024-05-27T07:01:09.259+0000 7ff5f0a00640 -1 mds.e *** got signal Terminated *** 2024-05-27T07:01:09.258 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-05-27T07:01:09.259+0000 7f8d2ce00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 27923) UID: 0 2024-05-27T07:01:09.258 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-05-27T07:01:09.259+0000 7f8d2ce00640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-27T07:01:09.258 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T07:01:09.259+0000 7fdcf0e00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 27928) UID: 0 2024-05-27T07:01:09.258 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-05-27T07:01:09.259+0000 7fdcf0e00640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-27T07:01:09.450 INFO:tasks.ceph.mgr.x.smithi106.stderr:daemon-helper: command crashed with signal 15 2024-05-27T07:01:09.983 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-27T07:01:10.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.147+0000 7f170fe00640 1 Processor -- start 2024-05-27T07:01:10.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.147+0000 7f170fe00640 1 -- start start 2024-05-27T07:01:10.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170fe00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f1708072430 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c5040 con 0x7f1708161cd0 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170fe00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f17080c5b60 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f1708072430 legacy=0x7f1708159b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 msgr2=0x7f17081640f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 msgr2=0x7f17080c5f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:10.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:10.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.151+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:10.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.355+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f1708072430 legacy=0x7f1708159b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:10.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.355+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 msgr2=0x7f17081640f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:10.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.355+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:10.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.355+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 msgr2=0x7f17080c5f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:10.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.355+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.759+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 msgr2=0x7f17081640f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.759+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f1708072430 legacy=0x7f1708159b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.759+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.759+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 msgr2=0x7f17080c5f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:10.759+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:11.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:11.563+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 msgr2=0x7f17081640f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:11.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:11.563+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f1708072430 legacy=0x7f1708159b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:11.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:11.563+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:11.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:11.563+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 msgr2=0x7f17080c5f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:11.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:11.563+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 msgr2=0x7f17080c5f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f17080c5b60 0x7f17080c5f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 msgr2=0x7f17081640f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1708161cd0 0x7f17081640f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f1708072430 legacy=0x7f1708159b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:13.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:13.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e80036d0 2024-05-27T07:01:13.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80073d0 2024-05-27T07:01:13.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:13.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:13.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.151+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:13.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.355+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:13.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.355+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:13.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.355+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:13.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.759+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:13.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.759+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:13.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:13.759+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:14.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:14.563+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:14.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:14.563+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:14.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:14.563+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:16.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80036d0 2024-05-27T07:01:16.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006350 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e80073d0 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:16.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.151+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:16.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.355+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:16.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.355+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:16.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.355+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:16.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.355+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:16.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.355+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.759+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.759+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.759+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.759+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:16.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:16.759+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:17.563+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:17.563+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:17.563+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:17.563+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:17.563+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:19.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 msgr2=0x7f16e80098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:19.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80073d0 0x7f16e80098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:19.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:19.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:19.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80036d0 legacy=0x7f16e8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8011c80 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e800f320 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8011c80 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:19.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.151+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:19.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.355+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8011c80 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:19.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.355+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:19.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.355+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:19.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.759+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8011c80 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:19.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.759+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:19.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:19.759+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:20.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:20.563+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8011c80 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:20.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:20.563+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:20.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:20.563+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:22.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:22.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:22.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8011c80 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:22.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:22.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:22.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:22.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8015960 2024-05-27T07:01:22.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006350 2024-05-27T07:01:22.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e800f320 2024-05-27T07:01:22.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:22.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:22.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:22.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:22.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.151+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:22.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.355+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:22.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.355+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:22.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.355+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:22.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.355+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:22.344 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.355+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.759+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:22.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.759+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.759+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.759+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:22.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:22.759+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:23.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:23.563+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:23.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:23.563+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:23.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:23.563+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:23.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:23.563+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:23.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:23.563+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:25.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:25.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800f320 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8009880 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8003ae0 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:25.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e80169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:25.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.151+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:25.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.355+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.355+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e80169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:25.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.355+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:25.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.759+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:25.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.759+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e80169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:25.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:25.759+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:26.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:26.563+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:26.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:26.563+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e80169e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:26.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:26.563+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:28.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e80169e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:28.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e80169e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:28.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8015960 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8009880 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006350 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:28.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:28.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:28.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.155+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:28.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.359+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:28.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.359+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:28.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.359+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:28.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.359+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:28.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.763+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:28.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.763+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:28.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.763+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:28.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:28.763+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:29.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:29.567+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:29.567+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:29.567+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:29.567+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:31.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8009880 legacy=0x7f16e80140a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8003ae0 2024-05-27T07:01:31.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.151+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e800fc70 2024-05-27T07:01:31.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.155+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:31.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.155+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 msgr2=0x7f16e8010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:31.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.155+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:31.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.155+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:31.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.359+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 msgr2=0x7f16e8010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:31.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.359+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:31.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.359+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:31.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.359+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:31.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.763+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:31.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.763+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 msgr2=0x7f16e8010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:31.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.763+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:31.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:31.763+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:32.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:32.567+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 msgr2=0x7f16e8010080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:32.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:32.567+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:32.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:32.567+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:32.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:32.567+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:34.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 msgr2=0x7f16e8010080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:34.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e800fc70 0x7f16e8010080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:34.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8003ae0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8015960 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006350 2024-05-27T07:01:34.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e80097f0 2024-05-27T07:01:34.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:34.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:34.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:34.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 msgr2=0x7f16e8014850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:34.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.155+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:34.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.359+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:34.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.359+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:34.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.359+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:34.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.359+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 msgr2=0x7f16e8014850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:34.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.359+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.763+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.763+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.763+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.763+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 msgr2=0x7f16e8014850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:34.763+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:35.316 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:01:35.320 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:01:35.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:35.567+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:35.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:35.567+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:35.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:35.567+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:35.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:35.567+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 msgr2=0x7f16e8014850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:35.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:35.567+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:37.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 msgr2=0x7f16e8014850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:37.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e80097f0 0x7f16e8014850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 msgr2=0x7f16e80067f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006350 0x7f16e80067f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8005af0 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80154a0 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:37.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:37.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.155+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:37.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.359+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:37.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.359+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:37.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.359+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:37.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.763+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:37.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.763+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:37.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:37.763+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:38.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:38.567+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:38.567+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8014d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:38.567+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:40.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8014d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8015960 2024-05-27T07:01:40.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80097f0 2024-05-27T07:01:40.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80154a0 2024-05-27T07:01:40.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:40.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:40.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:40.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.155+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:40.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.359+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:40.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.359+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:40.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.359+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:40.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.359+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:40.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.763+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.763+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.763+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:40.763+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:41.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:41.567+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:41.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:41.567+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:41.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:41.567+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:41.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:41.567+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:43.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 msgr2=0x7f16e8011740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:43.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80154a0 0x7f16e8011740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:43.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015960 legacy=0x7f16e8015d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80127a0 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8006350 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 msgr2=0x7f16e800e320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:43.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:43.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:43.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.155+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:43.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.359+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 msgr2=0x7f16e800e320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:43.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.359+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:43.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.359+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:43.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.359+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.763+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 msgr2=0x7f16e800e320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.763+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.763+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:43.763+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:44.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:44.567+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 msgr2=0x7f16e800e320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:44.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:44.567+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:44.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:44.567+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:44.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:44.567+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:46.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:46.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:46.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 msgr2=0x7f16e800e320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:46.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80127a0 0x7f16e800e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:46.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e80153b0 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e80162b0 con 0x7f16e8006350 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:46.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:46.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:46.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:46.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:46.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.155+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:46.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:46.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:46.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.359+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:46.763+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8009f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:47.567+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:49.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 msgr2=0x7f16e80067c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:49.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006350 0x7f16e80067c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:49.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:49.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8009f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8009f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8005af0 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e80097f0 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80169a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:49.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.155+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e800a4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:49.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.359+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80169a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:49.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.359+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e800a4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:49.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.763+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80169a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:49.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:49.763+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e800a4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:50.567+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80169a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:50.567+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e800a4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:52.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e800a4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:52.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80169a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e80153b0 2024-05-27T07:01:52.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e80162b0 con 0x7f16e8006850 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:52.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.155+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:52.346 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:52.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.359+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:52.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:52.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:52.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:52.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:52.763+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:53.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:01:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:01:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:53.567+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:01:55.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8006850 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8005af0 2024-05-27T07:01:55.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e80097f0 2024-05-27T07:01:55.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8016960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:55.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.159+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:55.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.363+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8016960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:55.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.363+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.767+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8016960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:55.767+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:56.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:56.571+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8016960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:56.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:56.571+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:58.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:58.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e8016960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:01:58.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:01:58.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:01:58.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8005af0 2024-05-27T07:01:58.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80097f0 2024-05-27T07:01:58.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80153b0 2024-05-27T07:01:58.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:58.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:58.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:58.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.159+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:01:58.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.363+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:58.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.363+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:58.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.363+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:58.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.363+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:01:58.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.767+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:58.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.767+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:58.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.767+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:58.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:58.767+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:01:59.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:59.571+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:01:59.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:59.571+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:01:59.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:59.571+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:01:59.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:01:59.571+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:01.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:02:01.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8011740 2024-05-27T07:02:01.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:01.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:01.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:01.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.159+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:01.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.363+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:01.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.363+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.363+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:01.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.363+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.767+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.767+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.767+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:01.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:01.767+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:02.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:02.571+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:02.571+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:02.571+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:02.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:02.571+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:04.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:04.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8005af0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e80097f0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80153b0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:04.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.158+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:04.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.362+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:04.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.362+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:04.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.362+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:04.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.362+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.766+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.766+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.766+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:04.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:04.766+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:05.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:02:05.380 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:02:05.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:05.570+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:05.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:05.570+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:05.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:05.570+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:05.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:05.570+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:07.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:07.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:02:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8011740 2024-05-27T07:02:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:07.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.158+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:07.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.362+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:07.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.362+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:07.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.362+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:07.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.362+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.766+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.766+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.766+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:07.766+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:08.570+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:08.570+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:08.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:08.570+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:08.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:08.570+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:10.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 msgr2=0x7f16e8006270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015ea0 0x7f16e8006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8005af0 2024-05-27T07:02:10.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e80097f0 2024-05-27T07:02:10.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8011740 2024-05-27T07:02:10.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:10.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:10.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:10.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.158+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:10.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.362+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:10.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.362+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:10.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.362+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:10.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.362+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:10.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.766+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:10.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.766+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:10.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.766+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:10.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:10.766+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:11.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:11.570+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:11.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:11.570+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:11.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:11.570+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:11.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:11.570+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:13.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80153b0 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:13.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.158+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:13.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.362+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:13.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.362+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:13.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.362+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:13.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.362+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:13.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.766+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:13.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.766+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:13.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.766+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:13.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:13.766+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:14.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:14.570+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:14.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:14.570+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:14.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:14.570+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:14.560 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:14.570+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:16.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:16.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 msgr2=0x7f16e8015960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80153b0 0x7f16e8015960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8005af0 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80097f0 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:02:16.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:16.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:16.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:16.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.158+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:16.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.362+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:16.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.362+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:16.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.362+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:16.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.362+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.766+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.766+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.766+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:16.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:16.766+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:17.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:17.570+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:17.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:17.570+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:17.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:17.570+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:17.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:17.570+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:19.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006800 2024-05-27T07:02:19.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8011740 2024-05-27T07:02:19.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:19.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:19.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:19.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.158+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:19.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.362+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:19.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.362+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:19.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.362+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:19.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.362+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.766+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.766+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.766+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:19.766+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:20.570+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:20.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:20.570+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:20.570+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:20.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:20.570+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:22.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:22.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8005af0 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e80097f0 2024-05-27T07:02:22.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006800 2024-05-27T07:02:22.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:22.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:22.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:22.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.158+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:22.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.362+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:22.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.362+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:22.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.362+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:22.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.362+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.766+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.766+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.766+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:22.766+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:23.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:23.570+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:23.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:23.570+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:23.558 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:23.570+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:23.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:23.570+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:25.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:25.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8011740 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:25.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.162+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:25.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.366+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:25.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.366+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:25.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.366+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:25.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.366+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.770+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:25.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.770+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.770+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:25.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:25.770+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:26.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:26.574+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:26.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:26.574+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:26.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:26.574+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:26.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:26.574+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:28.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:28.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:28.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80097f0 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8011740 2024-05-27T07:02:28.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:28.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:28.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:28.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:28.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:28.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.162+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:28.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.366+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:28.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.366+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:28.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.366+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:28.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.366+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:28.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.366+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:28.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.770+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:28.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.770+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:28.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.770+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:28.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.770+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:28.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:28.770+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:29.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:29.574+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:29.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:29.574+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:29.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:29.574+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:29.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:29.574+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:29.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:29.574+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:31.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:31.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:31.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:31.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8005af0 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8006800 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:31.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.162+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:31.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.366+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:31.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.366+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:31.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.366+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:31.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.770+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:31.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.770+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:31.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:31.770+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:32.574+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:32.574+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:32.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:32.574+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:34.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8006800 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e80097f0 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8011740 2024-05-27T07:02:34.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:34.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.162+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:34.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.366+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:34.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.366+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:34.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.366+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:34.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.366+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:34.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.366+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:34.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.770+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:34.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.770+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:34.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.770+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:34.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.770+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:34.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:34.770+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:35.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:02:35.453 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:02:35.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:35.574+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:35.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:35.574+0000 7f170de00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:35.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:35.574+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:35.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:35.574+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:35.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:35.574+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:37.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:37.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8006800 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8005af0 2024-05-27T07:02:37.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.162+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:37.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.366+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:37.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.366+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:37.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.366+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:37.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.770+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:37.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.770+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:37.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:37.770+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:38.574+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:38.574+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:38.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:38.574+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:40.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:40.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8005af0 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:40.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8006800 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8011740 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:40.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:40.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.162+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:40.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.366+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:40.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.366+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:40.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.366+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:40.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.366+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:40.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.366+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.770+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.770+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:40.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.770+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.770+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:40.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:40.770+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:41.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:41.574+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:41.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:41.574+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:41.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:41.574+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:41.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:41.574+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:41.563 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:41.574+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:43.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8011740 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e80097f0 2024-05-27T07:02:43.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015400 2024-05-27T07:02:43.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:43.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:43.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.162+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:43.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.366+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:43.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.366+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:43.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.366+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:43.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.770+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:43.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.770+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:43.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:43.770+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:44.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:44.574+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:44.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:44.574+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:44.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:44.574+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:46.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8015920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:46.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015400 0x7f16e8015920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:46.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e80097f0 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8015400 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17080c5270 con 0x7f16e8017fc0 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015ea0 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015400 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:46.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e801a3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:46.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.162+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:46.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.366+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015400 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.366+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.366+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e801a3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:46.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.366+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.770+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015400 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.770+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.770+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e801a3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:46.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:46.770+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:47.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:47.574+0000 7f170de00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015400 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:47.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:47.574+0000 7f170ca00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:47.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:47.574+0000 7f170d400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e801a3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:47.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:47.574+0000 7f170ca00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:49.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e801a3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8015400 legacy=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80158c0 2024-05-27T07:02:49.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8006800 2024-05-27T07:02:49.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:49.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:49.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e80127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:49.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.162+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.366+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.366+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.366+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e80127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:49.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.366+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:49.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.770+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:49.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.770+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:49.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.770+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e80127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:49.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:49.770+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:50.574+0000 7f170ca00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:50.574+0000 7f170ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:50.574+0000 7f170d400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e80127a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:50.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:50.574+0000 7f170d400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:52.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 msgr2=0x7f16e80127a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:52.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8006800 0x7f16e80127a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:52.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:52.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:52.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8017fc0 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80158c0 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:52.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:52.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e8016de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:52.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:52.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.166+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:52.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.370+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:52.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.370+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:52.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.370+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e8016de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:52.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.370+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:52.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.370+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.774+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.774+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.774+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e8016de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.774+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:52.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:52.774+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:53.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:53.578+0000 7f170de00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:02:53.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:53.578+0000 7f170de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:53.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:53.578+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:53.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:53.578+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e8016de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:02:53.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:53.578+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:55.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:55.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:55.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e80168a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:55.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e80168a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8017fc0 legacy=0x7f16e8016de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8006800 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015400 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80174f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:55.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.166+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:55.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.370+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80174f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:55.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.370+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:55.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.370+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:55.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.774+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80174f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:55.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.774+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:55.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:55.774+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:56.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:56.578+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80174f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:56.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:56.578+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:56.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:56.578+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:58.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80174f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:02:58.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8006800 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015400 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:58.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.166+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:02:58.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.370+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:58.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.370+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:58.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.370+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:58.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.370+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:58.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.370+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:02:58.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.774+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:58.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.774+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:58.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.774+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:58.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.774+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:58.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:58.774+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:02:59.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:59.578+0000 7f170ca00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:02:59.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:59.578+0000 7f170d400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:02:59.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:59.578+0000 7f170de00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:02:59.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:59.578+0000 7f170de00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:02:59.567 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:02:59.578+0000 7f170d400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:01.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:01.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e8015400 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8005af0 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80158c0 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:01.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:01.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.166+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:01.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.370+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:01.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.370+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:01.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.370+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:01.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.774+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:01.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.774+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:01.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:01.774+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:02.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:02.578+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:02.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:02.578+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:02.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:02.578+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:04.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e80158c0 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170de00640 1 -- reap_dead start 2024-05-27T07:03:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:04.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e8005f10 con 0x7f16e8006800 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e80158c0 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f1706a00640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8006f30 con 0x7f16e8015ea0 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:04.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.166+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:04.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.370+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:04.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.370+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:04.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.370+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:04.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.370+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:04.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.370+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:04.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.774+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:04.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.774+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:04.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.774+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:04.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.774+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:04.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:04.774+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:05.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:03:05.512 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:03:05.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:05.578+0000 7f170d400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:05.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:05.578+0000 7f170ca00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:05.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:05.578+0000 7f170de00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:05.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:05.578+0000 7f170de00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:05.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:05.578+0000 7f170ca00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 msgr2=0x7f16e8006240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f16e8015ea0 0x7f16e8006240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 msgr2=0x7f16e8012090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f16e80158c0 0x7f16e8012090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:07.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f16e8006800 legacy=0x7f16e80125d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 0x7f16e800f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1708072840 con 0x7f16e8005af0 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f1706a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f17080c75c0 con 0x7f16e8015280 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:07.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.166+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:07.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.370+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:07.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.370+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:07.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.370+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.774+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.774+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:07.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:07.774+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:08.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:08.578+0000 7f170d400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 msgr2=0x7f16e800f730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:08.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:08.578+0000 7f170d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f16e8015280 0x7f16e800f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:08.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:08.578+0000 7f170ca00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f16e8005af0 legacy=0x7f16e800fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-27T07:03:10.012 ERROR:tasks.mds_thrash.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/mds_thrash.py", line 121, in _run self.do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/mds_thrash.py", line 304, in do_thrash status = self.wait_for_stable(rank, gid) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/mds_thrash.py", line 182, in wait_for_stable status = self.fs.status() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/cephfs/filesystem.py", line 413, in status return FSStatus(self.mon_manager, epoch) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/cephfs/filesystem.py", line 88, in __init__ self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-05-27T07:03:10.029 INFO:tasks.mds_thrash:done joining 2024-05-27T07:03:10.030 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-27T07:03:10.041 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-27T07:03:10.041 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-05-27T07:03:10.041 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-27T07:03:10.064 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350296k,nr_inodes=4087574,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5190 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-05-27T07:03:10.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:03:10.064 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-05-27T07:03:10.113 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-27T07:03:10.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:03:10.115 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-27T07:03:10.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-27T07:03:10.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-05-27T07:03:10.380 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-27T07:03:10.390 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-27T07:03:10.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-27T07:03:10.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 Processor -- start 2024-05-27T07:03:10.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 -- start start 2024-05-27T07:03:10.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fe00cfdb0 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fe00ca770 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fe5200640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fe00d01c0 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fe00cfdb0 legacy=0x7f0fe0154200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fe00ca770 legacy=0x7f0fe00cb7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:10.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fe0165d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:10.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.550+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:10.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.754+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fe00cfdb0 legacy=0x7f0fe0154200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:10.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.754+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fe0165d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:10.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.754+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fe00ca770 legacy=0x7f0fe00cb7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:10.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:10.754+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:11.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.158+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fe00cfdb0 legacy=0x7f0fe0154200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:11.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.158+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fe0165d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:11.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.158+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:11.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.158+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fe00ca770 legacy=0x7f0fe00cb7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:11.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.962+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fe00cfdb0 legacy=0x7f0fe0154200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:11.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.962+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fe00ca770 legacy=0x7f0fe00cb7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:11.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.962+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fe0165d80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:11.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:11.962+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:13.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fe0165d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:13.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fe0165d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:13.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fe00ca770 legacy=0x7f0fe00cb7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:13.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fe00cfdb0 legacy=0x7f0fe0154200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:13.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:13.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:13.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0006140 2024-05-27T07:03:13.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc00036b0 2024-05-27T07:03:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 msgr2=0x7f0fc0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.554+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:13.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.758+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:13.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.758+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:13.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.758+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 msgr2=0x7f0fc0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:13.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:13.758+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:14.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.162+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:14.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.162+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:14.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.162+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 msgr2=0x7f0fc0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:14.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.162+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.966+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc0008650 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.966+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.966+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 msgr2=0x7f0fc0005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:14.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:14.966+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:16.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 msgr2=0x7f0fc0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00036b0 0x7f0fc0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc0008650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc0008650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:16.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0006140 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fe00d01c0 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc00087b0 con 0x7f0fc0009110 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc000bb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:16.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:16.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:16.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:16.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.554+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:16.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc000bb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:16.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:16.758+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc000bb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:17.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.162+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:17.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc000bb90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:17.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:17.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:17.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:17.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:17.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:17.966+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:19.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009110 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 msgr2=0x7f0fc000bb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0006140 0x7f0fc000bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:19.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0014020 2024-05-27T07:03:19.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc00104e0 2024-05-27T07:03:19.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014020 legacy=0x7f0fc00164d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:19.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.553+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0012900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:19.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.757+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014020 legacy=0x7f0fc00164d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:19.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:19.757+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0012900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:20.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:20.161+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014020 legacy=0x7f0fc00164d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:20.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:20.161+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0012900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:20.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:20.965+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014020 legacy=0x7f0fc00164d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:20.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:20.965+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0012900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:22.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0012900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:22.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014020 legacy=0x7f0fc00164d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc00104e0 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fe00d01c0 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000bb90 2024-05-27T07:03:22.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:22.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:22.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:22.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:22.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.553+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.757+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.757+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.757+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.757+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:22.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:22.757+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:23.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.161+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:23.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.161+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:23.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.161+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:23.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.161+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:23.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.161+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:23.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.965+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:23.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.965+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:23.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.965+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:23.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.965+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:23.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:23.965+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:25.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fe00d01c0 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:25.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0018440 2024-05-27T07:03:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0018850 2024-05-27T07:03:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:25.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.553+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:25.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.757+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:25.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.757+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:25.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:25.757+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:26.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.161+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:26.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.161+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:26.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.161+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:26.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.965+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:26.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.965+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:26.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:26.965+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:28.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 msgr2=0x7f0fc000b5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:28.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0018850 0x7f0fc000b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:28.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0018440 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc00104e0 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000bb90 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:28.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:28.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.553+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:28.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.757+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:28.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.757+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:28.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.757+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:28.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:28.757+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:29.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.161+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:29.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.161+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:29.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.161+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:29.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.161+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:29.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.965+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:29.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.965+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:29.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.965+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:29.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:29.965+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:31.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000b680 2024-05-27T07:03:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0016750 2024-05-27T07:03:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:31.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.553+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.757+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.757+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.757+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:31.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:31.757+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.161+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.161+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.161+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.161+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:32.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.965+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:32.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.965+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:32.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.965+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:32.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:32.965+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0016750 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:34.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0018440 2024-05-27T07:03:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc00104e0 2024-05-27T07:03:34.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:03:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:34.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.553+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.757+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.757+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.757+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:34.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:34.757+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:35.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.161+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:35.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.161+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:35.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.161+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:35.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.161+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:35.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:03:35.567 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:03:35.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.965+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:35.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.965+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:35.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.965+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:35.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:35.965+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:37.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:37.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:37.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000b680 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000bb90 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:37.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:37.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.553+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:37.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.757+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:37.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.757+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:37.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.757+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:37.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:37.757+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:38.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.161+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:38.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.161+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:38.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.161+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:38.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.161+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:38.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.965+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:38.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.965+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:38.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.965+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:38.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:38.965+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:40.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000b680 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0018440 2024-05-27T07:03:40.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc00104e0 2024-05-27T07:03:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:03:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.557+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:40.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.761+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:40.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.761+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:40.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.761+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:40.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:40.761+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:41.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.165+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:41.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.165+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:41.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.165+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:41.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.165+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.969+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.969+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:41.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.969+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:41.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:41.969+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:43.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000af00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:43.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:43.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc00104e0 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:43.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:43.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000bb90 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:43.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.557+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.761+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.761+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:43.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.761+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:43.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:43.761+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:44.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.165+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:44.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.165+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:44.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.165+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:44.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.165+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:44.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.969+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:44.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.969+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:44.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.969+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:44.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:44.969+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 msgr2=0x7f0fc00173b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:46.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000bb90 0x7f0fc00173b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a5b0 0x7f0fc000a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0018440 2024-05-27T07:03:46.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000baf0 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 msgr2=0x7f0fc0017390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:46.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.557+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.761+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.761+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:46.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.761+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.761+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 msgr2=0x7f0fc0017390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:46.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:46.761+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:47.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.165+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:47.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.165+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:47.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.165+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:47.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.165+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 msgr2=0x7f0fc0017390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:47.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.165+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:47.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.969+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:47.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.969+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:47.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.969+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:47.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.969+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 msgr2=0x7f0fc0017390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:47.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:47.969+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 msgr2=0x7f0fc0017390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000baf0 0x7f0fc0017390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:49.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0017d10 2024-05-27T07:03:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc00114f0 2024-05-27T07:03:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.553+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:49.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.757+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:49.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.757+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:49.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:49.757+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:50.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.161+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:50.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.161+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:50.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.161+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:50.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.965+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:50.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.965+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:50.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:50.965+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:52.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:52.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0018440 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc00114f0 2024-05-27T07:03:52.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:52.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:52.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:52.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:52.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.557+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.761+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.761+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.761+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:52.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.761+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:52.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:52.761+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:53.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.165+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:53.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.165+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:53.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.165+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:53.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.165+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:53.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.165+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.969+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:03:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.969+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.969+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:03:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.969+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:53.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:53.969+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:55.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000aca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:55.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0013910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0013910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0018440 legacy=0x7f0fc0016c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc000bad0 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0017390 2024-05-27T07:03:55.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:55.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc000bad0 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:55.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.557+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.761+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.761+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc000bad0 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:55.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:55.761+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:56.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.165+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc000bad0 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:56.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.165+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:56.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.165+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.969+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.969+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc000bad0 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:56.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:56.969+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:58.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc000bad0 legacy=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:03:58.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc001ad10 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0017390 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc00114f0 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc001ad10 legacy=0x7f0fc001d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:58.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.556+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:03:58.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.760+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc001ad10 legacy=0x7f0fc001d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:58.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.760+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:58.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.760+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:58.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.760+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:58.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:58.760+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:03:59.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.164+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc001ad10 legacy=0x7f0fc001d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:59.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.164+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:59.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.164+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:59.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.164+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:59.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.164+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:03:59.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.968+0000 7f0fde800640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:03:59.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.968+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:03:59.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.968+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc001ad10 legacy=0x7f0fc001d130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:03:59.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.968+0000 7f0fde800640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:03:59.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:03:59.968+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:01.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:01.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:01.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc001ad10 legacy=0x7f0fc001d130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0016c80 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0016c80 legacy=0x7f0fc000aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:01.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.556+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:01.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.760+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:01.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.760+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0016c80 legacy=0x7f0fc000aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:01.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:01.760+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:02.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.164+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:02.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.164+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0016c80 legacy=0x7f0fc000aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:02.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.164+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:02.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.968+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0016c80 legacy=0x7f0fc000aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:02.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.968+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:02.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:02.968+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:04.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0016c80 legacy=0x7f0fc000aad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0009580 2024-05-27T07:04:04.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0017390 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0009580 legacy=0x7f0fc0009990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:04.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.556+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:04.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.760+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:04.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.760+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0009580 legacy=0x7f0fc0009990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:04.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.760+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:04.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.760+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:04.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:04.760+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.164+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0009580 legacy=0x7f0fc0009990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.164+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.164+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:05.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.164+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:05.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.164+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:05.644 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:04:05.648 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:04:05.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.968+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:05.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.968+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0009580 legacy=0x7f0fc0009990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:05.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.968+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:05.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.968+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:05.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:05.968+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017390 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0009580 legacy=0x7f0fc0009990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:07.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:07.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0013910 2024-05-27T07:04:07.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc00114f0 2024-05-27T07:04:07.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0013910 legacy=0x7f0fc000a0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:07.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:07.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.556+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:07.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.760+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0013910 legacy=0x7f0fc000a0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:07.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.760+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:07.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:07.760+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.164+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.164+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0013910 legacy=0x7f0fc000a0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:08.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.164+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.968+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.968+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0013910 legacy=0x7f0fc000a0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:08.968+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:10.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 msgr2=0x7f0fc000a7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:10.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc00114f0 0x7f0fc000a7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0013910 legacy=0x7f0fc000a0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:10.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0017090 2024-05-27T07:04:10.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0009550 2024-05-27T07:04:10.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000acf0 2024-05-27T07:04:10.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:10.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc001cd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:10.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.556+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:10.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.760+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:10.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.760+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:10.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.760+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc001cd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:10.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:10.760+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.164+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.164+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc001cd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.164+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:11.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.164+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:11.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.968+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:11.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.968+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:11.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.968+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc001cd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:11.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:11.968+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:13.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc001cd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc001cd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:13.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0013910 2024-05-27T07:04:13.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000a070 2024-05-27T07:04:13.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc0015440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:13.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:13.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 msgr2=0x7f0fc000a480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:13.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.560+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:13.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.764+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc0015440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:13.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.764+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:13.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.764+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 msgr2=0x7f0fc000a480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:13.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:13.764+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:14.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.168+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc0015440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:14.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.168+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:14.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.168+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 msgr2=0x7f0fc000a480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:14.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.168+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:14.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.972+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc0015440 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:14.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.972+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:14.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.972+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 msgr2=0x7f0fc000a480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:14.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:14.972+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 msgr2=0x7f0fc000a480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc000a070 0x7f0fc000a480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc0015440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc0015440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:16.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0017090 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0009550 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000acf0 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:16.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.560+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.764+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.764+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.764+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:16.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:16.764+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:17.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.168+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:17.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.168+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:17.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.168+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:17.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.168+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:17.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.972+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:17.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.972+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:17.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.972+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:17.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:17.972+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:19.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0013910 2024-05-27T07:04:19.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0015440 2024-05-27T07:04:19.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:19.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:19.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc0015ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:19.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.560+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.764+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.764+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:19.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.764+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc0015ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:19.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:19.764+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:20.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.168+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:20.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.168+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:20.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.168+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc0015ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:20.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.168+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.972+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.972+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.972+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc0015ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:20.972+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:22.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc0015ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc0015ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0013910 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:22.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0009550 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc000acf0 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000a5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:22.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:22.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.560+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.764+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000a5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.764+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:22.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.764+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:22.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.764+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:22.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:22.764+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:23.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.168+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000a5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:23.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.168+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:23.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.168+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:23.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.168+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:23.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.168+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:23.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.972+0000 7f0fdd400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000a5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:23.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.972+0000 7f0fdd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:23.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.972+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:23.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.972+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:23.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:23.972+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:25.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc000a5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc000a5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:25.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0015e30 2024-05-27T07:04:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0015440 2024-05-27T07:04:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015e30 legacy=0x7f0fc0017090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:25.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.560+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:25.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.764+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015e30 legacy=0x7f0fc0017090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:25.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.764+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:25.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:25.764+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:26.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.168+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015e30 legacy=0x7f0fc0017090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:26.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.168+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:26.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.168+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:26.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.972+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015e30 legacy=0x7f0fc0017090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:26.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.972+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:26.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:26.972+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015440 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015e30 legacy=0x7f0fc0017090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:28.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0017d10 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0009550 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000acf0 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc001ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:28.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.560+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:28.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.764+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc001ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:28.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.764+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:28.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.764+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:28.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:28.764+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:29.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.168+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc001ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:29.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.168+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:29.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.168+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:29.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.168+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:29.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.972+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc001ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:29.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.972+0000 7f0fde800640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:29.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.972+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:29.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:29.972+0000 7f0fde800640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 msgr2=0x7f0fc0015850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc000acf0 0x7f0fc0015850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0009550 legacy=0x7f0fc0009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc001ed30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0005ad0 2024-05-27T07:04:31.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0017090 2024-05-27T07:04:31.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:31.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:31.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:31.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.560+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:31.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.764+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:31.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.764+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:31.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.764+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:31.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:31.764+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:32.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.168+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:32.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.168+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:32.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.168+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:32.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.168+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:32.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.972+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:32.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.972+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:32.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.972+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:32.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:32.972+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0017090 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc0005ad0 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:34.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0017d10 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0017090 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:34.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.560+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.764+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.764+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.764+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:34.764+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:35.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.168+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:35.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.168+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:35.154 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.168+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:35.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.168+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:35.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:04:35.710 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:04:35.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.972+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:35.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.972+0000 7f0fde800640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:35.958 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.972+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:35.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:35.972+0000 7f0fdd400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:37.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:37.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0015980 2024-05-27T07:04:37.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.560+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0009c60 2024-05-27T07:04:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.564+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.564+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.564+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:37.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.564+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:37.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.768+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:37.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.768+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:37.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.768+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:37.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:37.768+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.172+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.172+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.172+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:38.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.172+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:38.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.975+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:38.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.975+0000 7f0fdde00640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:38.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.975+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:38.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:38.975+0000 7f0fdde00640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:40.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 msgr2=0x7f0fc000a070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0009c60 0x7f0fc000a070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:40.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0017090 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0015980 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:40.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.563+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:40.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.767+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:40.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.767+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:40.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.767+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:40.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.767+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:40.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:40.767+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:41.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.171+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:41.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.171+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:41.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.171+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:41.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.171+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:41.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.171+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.975+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.975+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:41.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.975+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.975+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:41.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:41.975+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 msgr2=0x7f0fc0018120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0015980 0x7f0fc0018120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:43.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:43.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0014eb0 2024-05-27T07:04:43.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0017d10 2024-05-27T07:04:43.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014eb0 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:43.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 msgr2=0x7f0fc0014970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:43.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.563+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:43.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.767+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014eb0 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:43.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.767+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 msgr2=0x7f0fc0014970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:43.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:43.767+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:44.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.171+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014eb0 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:44.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.171+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 msgr2=0x7f0fc0014970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:44.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.171+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:44.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.975+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014eb0 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:44.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.975+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 msgr2=0x7f0fc0014970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:44.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:44.975+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:46.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 msgr2=0x7f0fc0014970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:46.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0017d10 0x7f0fc0014970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0014eb0 legacy=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc00152c0 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0017090 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:46.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:46.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:46.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.563+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:46.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.767+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:46.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.767+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:46.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.767+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:46.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:46.767+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:47.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.171+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:47.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.171+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:47.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.171+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:47.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.171+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:47.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.975+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:47.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.975+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:47.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.975+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:47.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:47.975+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:49.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:49.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:49.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc00156d0 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0014970 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:49.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:49.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.563+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:49.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.767+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:49.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.767+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:49.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.767+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:49.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:49.767+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:50.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.171+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:50.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.171+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:50.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.171+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:50.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.171+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:50.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.975+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:50.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.975+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:50.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.975+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:50.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:50.975+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:52.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc00152c0 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c78b0 con 0x7f0fc0017090 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:52.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.563+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:52.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.767+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:52.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.767+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:52.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.767+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:52.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:52.767+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:53.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.171+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:53.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.171+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:53.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.171+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:53.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.171+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:53.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.975+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:04:53.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.975+0000 7f0fde800640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:53.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.975+0000 7f0fde800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:53.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:53.975+0000 7f0fdde00640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc00152c0 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc00156d0 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0014970 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.563+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.767+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:55.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.767+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.767+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:55.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:55.767+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:56.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.171+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:56.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.171+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:56.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.171+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:56.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.171+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:56.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.975+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:56.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.975+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:04:56.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.975+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:56.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:56.975+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:58.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 msgr2=0x7f0fc0017d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc00156d0 0x7f0fc0017d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0017090 2024-05-27T07:04:58.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0014970 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:58.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.563+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:04:58.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.767+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:58.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.767+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:58.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.767+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:58.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.767+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:58.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:58.767+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:04:59.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.171+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:59.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.171+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:59.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.171+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:59.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.171+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:59.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.171+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:04:59.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.975+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:04:59.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.975+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:04:59.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.975+0000 7f0fdd400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:04:59.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.975+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:04:59.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:04:59.975+0000 7f0fdd400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc0009550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc0009550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:01.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0017d10 2024-05-27T07:05:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0018120 2024-05-27T07:05:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:01.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.563+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:05:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.767+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.767+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:01.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:01.767+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:05:02.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.171+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:02.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.171+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:02.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.171+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:05:02.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.975+0000 7f0fdde00640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:02.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.975+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:02.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:02.975+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:05:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:04.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0017d10 legacy=0x7f0fc0009a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fde800640 1 -- reap_dead start 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c6ee0 con 0x7f0fc0015800 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 -- --> v1:172.21.15.106:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe00c7b70 con 0x7f0fc0017090 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc000a5b0 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:05:04.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:05:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015800 legacy=0x7f0fc0012910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:04.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.563+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:05:04.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.767+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:05:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.767+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015800 legacy=0x7f0fc0012910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.767+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:05:04.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:04.767+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:05:05.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.171+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:05:05.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.171+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:05:05.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.171+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015800 legacy=0x7f0fc0012910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:05.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.171+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:05:05.763 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:05.766 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:05.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.975+0000 7f0fdde00640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.81:3300/0 2024-05-27T07:05:05.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.975+0000 7f0fdd400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015800 legacy=0x7f0fc0012910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-05-27T07:05:05.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.975+0000 7f0fdde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:05:05.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:05.975+0000 7f0fde800640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.106:6789/0 2024-05-27T07:05:07.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 -- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 msgr2=0x7f0fc0014430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fc000a5b0 0x7f0fc0014430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.106:6789/0 conn(0x7f0fc0017090 legacy=0x7f0fc000a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7f0fc0015800 legacy=0x7f0fc0012910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc00152c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc0013020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-27T07:05:07.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 -- --> v2:172.21.15.106:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe00c71c0 con 0x7f0fc0018120 2024-05-27T07:05:07.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fd7400640 1 -- --> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0008d00 con 0x7f0fc0014970 2024-05-27T07:05:07.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:05:07.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc0013020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:07.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:05:07.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.563+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc0013020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-27T07:05:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.767+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:05:07.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.767+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc0013020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.767+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:05:07.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:07.767+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc0013020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-27T07:05:08.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.171+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:05:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.171+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc0013020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.171+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:05:08.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.171+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc0013020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-27T07:05:08.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.975+0000 7f0fdd400640 1 -- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 msgr2=0x7f0fc0013020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3300/0 2024-05-27T07:05:08.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.975+0000 7f0fdde00640 1 -- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 msgr2=0x7f0fc00152c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.106:3301/0 2024-05-27T07:05:08.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.975+0000 7f0fdd400640 1 --2- >> v2:172.21.15.106:3300/0 conn(0x7f0fc0018120 0x7f0fc0013020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:05:08.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-05-27T07:05:08.975+0000 7f0fdde00640 1 --2- >> [v2:172.21.15.106:3301/0,v1:172.21.15.106:6790/0] conn(0x7f0fc0014970 0x7f0fc00152c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-27T07:05:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-27T07:05:10.418 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-05-27T07:05:10.429 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-27T07:05:10.429 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-27T07:05:10.429 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:10.432 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-05-27T07:05:10.442 INFO:tasks.ceph.mds.a:Stopped 2024-05-27T07:05:10.442 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-27T07:05:10.442 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.442 INFO:tasks.ceph.mds.c:Stopped 2024-05-27T07:05:10.443 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-05-27T07:05:10.443 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.443 INFO:tasks.ceph.mds.e:Stopped 2024-05-27T07:05:10.443 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-27T07:05:10.443 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.443 INFO:tasks.ceph.mds.b:Stopped 2024-05-27T07:05:10.443 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-27T07:05:10.444 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.444 INFO:tasks.ceph.mds.d:Stopped 2024-05-27T07:05:10.444 ERROR:tasks.ceph.mds.f:tried to stop a non-running daemon 2024-05-27T07:05:10.446 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-27T07:05:10.446 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-27T07:05:10.446 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.446 INFO:tasks.ceph.osd.0:Stopped 2024-05-27T07:05:10.446 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-27T07:05:10.447 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.447 INFO:tasks.ceph.osd.1:Stopped 2024-05-27T07:05:10.447 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-27T07:05:10.447 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.447 INFO:tasks.ceph.osd.2:Stopped 2024-05-27T07:05:10.447 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-27T07:05:10.447 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.448 INFO:tasks.ceph.osd.3:Stopped 2024-05-27T07:05:10.448 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-27T07:05:10.448 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.448 INFO:tasks.ceph.osd.4:Stopped 2024-05-27T07:05:10.448 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-27T07:05:10.448 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.449 INFO:tasks.ceph.osd.5:Stopped 2024-05-27T07:05:10.449 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-27T07:05:10.449 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.449 INFO:tasks.ceph.osd.6:Stopped 2024-05-27T07:05:10.449 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-27T07:05:10.449 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.449 INFO:tasks.ceph.osd.7:Stopped 2024-05-27T07:05:10.450 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-27T07:05:10.450 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-27T07:05:10.450 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.450 INFO:tasks.ceph.mgr.y:Stopped 2024-05-27T07:05:10.450 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-27T07:05:10.450 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:10.451 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi106 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2024-05-27T07:05:10.461 INFO:tasks.ceph.mgr.x:Stopped 2024-05-27T07:05:10.461 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-27T07:05:10.461 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-27T07:05:10.461 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.462 INFO:tasks.ceph.mon.a:Stopped 2024-05-27T07:05:10.462 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-27T07:05:10.462 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.464 INFO:tasks.ceph.mon.b:Stopped 2024-05-27T07:05:10.464 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-27T07:05:10.464 INFO:teuthology.orchestra.run:waiting for 300 2024-05-27T07:05:10.464 INFO:tasks.ceph.mon.c:Stopped 2024-05-27T07:05:10.465 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-27T07:05:10.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'Replacing daemon mds' | head -n 1 2024-05-27T07:05:10.524 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:05:10.524 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-27T07:05:10.783 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:05:10.783 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-27T07:05:10.998 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:05:10.998 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-27T07:05:11.169 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:05:11.169 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-27T07:05:11.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T07:05:11.335 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-27T07:05:11.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T07:05:11.571 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-27T07:05:11.780 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T07:05:11.781 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-27T07:05:11.955 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T07:05:11.955 DEBUG:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-27T07:05:12.149 INFO:tasks.ceph:Archiving mon data... 2024-05-27T07:05:12.150 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/data/mon.a.tgz 2024-05-27T07:05:12.151 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-05-27T07:05:12.160 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.Qya52j3eqj 2024-05-27T07:05:12.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Qya52j3eqj 2024-05-27T07:05:12.279 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.Qya52j3eqj 2024-05-27T07:05:12.354 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.Qya52j3eqj is 282KB 2024-05-27T07:05:12.416 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.Qya52j3eqj 2024-05-27T07:05:12.424 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/data/mon.b.tgz 2024-05-27T07:05:12.425 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-05-27T07:05:12.433 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.lXflqbzuWI 2024-05-27T07:05:12.434 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.lXflqbzuWI 2024-05-27T07:05:12.552 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.lXflqbzuWI 2024-05-27T07:05:12.631 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.lXflqbzuWI is 279KB 2024-05-27T07:05:12.693 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.lXflqbzuWI 2024-05-27T07:05:12.701 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/data/mon.c.tgz 2024-05-27T07:05:12.701 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-05-27T07:05:12.750 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.4JU92tVVjx 2024-05-27T07:05:12.751 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4JU92tVVjx 2024-05-27T07:05:12.865 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.4JU92tVVjx 2024-05-27T07:05:12.946 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.4JU92tVVjx is 283KB 2024-05-27T07:05:13.013 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.4JU92tVVjx 2024-05-27T07:05:13.021 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-27T07:05:13.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-27T07:05:13.024 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-27T07:05:13.093 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-27T07:05:13.131 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-27T07:05:13.131 INFO:tasks.ceph:Archiving crash dumps... 2024-05-27T07:05:13.133 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi081/crash 2024-05-27T07:05:13.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-27T07:05:13.165 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi106/crash 2024-05-27T07:05:13.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-27T07:05:13.192 INFO:tasks.ceph:Shutting down logrotate 2024-05-27T07:05:13.193 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:13.204 DEBUG:teuthology.orchestra.run.smithi106:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:13.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:13.297 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-27T07:05:13.337 INFO:tasks.ceph:Compressing logs... 2024-05-27T07:05:13.338 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:05:13.347 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-27T07:05:13.361 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35240.log 2024-05-27T07:05:13.362 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-05-27T07:05:13.362 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33514.log 2024-05-27T07:05:13.363 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35240.log.gz 2024-05-27T07:05:13.364 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-05-27T07:05:13.364 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-05-27T07:05:13.364 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-05-27T07:05:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41723.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.33514.log.gz 2024-05-27T07:05:13.365 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32712.log 2024-05-27T07:05:13.365 INFO:teuthology.orchestra.run.smithi081.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-05-27T07:05:13.366 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34056.log 2024-05-27T07:05:13.366 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-05-27T07:05:13.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-05-27T07:05:13.367 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32712.log.gz 2024-05-27T07:05:13.367 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-05-27T07:05:13.368 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34056.log.gz 2024-05-27T07:05:13.368 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35680.log 2024-05-27T07:05:13.368 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-05-27T07:05:13.369 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38391.log 2024-05-27T07:05:13.369 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-05-27T07:05:13.370 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38247.log 2024-05-27T07:05:13.370 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35680.log.gz 2024-05-27T07:05:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33847.log 2024-05-27T07:05:13.372 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38391.log.gz 2024-05-27T07:05:13.372 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38247.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-27T07:05:13.372 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38247.log.gz 2024-05-27T07:05:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.0.33847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39132.log 2024-05-27T07:05:13.373 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-05-27T07:05:13.374 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34631.log 2024-05-27T07:05:13.375 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39132.log.gz 2024-05-27T07:05:13.375 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35741.log 2024-05-27T07:05:13.376 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-05-27T07:05:13.376 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-05-27T07:05:13.376 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34631.log.gz 2024-05-27T07:05:13.377 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37138.log 2024-05-27T07:05:13.378 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35741.log.gz 2024-05-27T07:05:13.379 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-05-27T07:05:13.379 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-05-27T07:05:13.379 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37363.log 2024-05-27T07:05:13.380 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37138.log.gz 2024-05-27T07:05:13.380 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-05-27T07:05:13.381 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-05-27T07:05:13.382 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37363.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37363.log.gz 2024-05-27T07:05:13.382 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32051.log 2024-05-27T07:05:13.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2024-05-27T07:05:13.384 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-05-27T07:05:13.384 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40615.log 2024-05-27T07:05:13.384 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32051.log.gz 2024-05-27T07:05:13.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32025.log 2024-05-27T07:05:13.386 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2024-05-27T07:05:13.386 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40615.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-27T07:05:13.386 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40615.log.gz 2024-05-27T07:05:13.387 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31877.log 2024-05-27T07:05:13.387 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32025.log.gz 2024-05-27T07:05:13.387 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2024-05-27T07:05:13.388 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42719.log 2024-05-27T07:05:13.388 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31877.log.gz 2024-05-27T07:05:13.389 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31913.log 2024-05-27T07:05:13.389 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2024-05-27T07:05:13.390 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32913.log 2024-05-27T07:05:13.391 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42719.log.gz 2024-05-27T07:05:13.391 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-05-27T07:05:13.392 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31913.log.gz 2024-05-27T07:05:13.392 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-05-27T07:05:13.393 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32913.log.gz 2024-05-27T07:05:13.393 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-05-27T07:05:13.394 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-05-27T07:05:13.395 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-05-27T07:05:13.395 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-05-27T07:05:13.396 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41592.log 2024-05-27T07:05:13.396 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-05-27T07:05:13.397 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39750.log 2024-05-27T07:05:13.397 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-05-27T07:05:13.398 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41540.log 2024-05-27T07:05:13.398 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2024-05-27T07:05:13.399 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32705.log 2024-05-27T07:05:13.399 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39750.log.gz 2024-05-27T07:05:13.400 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28149.log 2024-05-27T07:05:13.400 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32705.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41540.log.gz 2024-05-27T07:05:13.401 INFO:teuthology.orchestra.run.smithi081.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32705.log.gz 2024-05-27T07:05:13.401 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-05-27T07:05:13.402 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-05-27T07:05:13.402 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28149.log.gz 2024-05-27T07:05:13.403 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34579.log 2024-05-27T07:05:13.403 INFO:teuthology.orchestra.run.smithi081.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-05-27T07:05:13.404 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39672.log 2024-05-27T07:05:13.404 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-05-27T07:05:13.405 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40935.log 2024-05-27T07:05:13.406 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34579.log.gz 2024-05-27T07:05:13.406 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-05-27T07:05:13.407 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39672.log.gz 2024-05-27T07:05:13.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-05-27T07:05:13.408 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40935.log.gz 2024-05-27T07:05:13.408 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38801.log 2024-05-27T07:05:13.409 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-05-27T07:05:13.410 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32501.log 2024-05-27T07:05:13.410 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-05-27T07:05:13.411 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38116.log 2024-05-27T07:05:13.411 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38801.log.gz 2024-05-27T07:05:13.412 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42797.log 2024-05-27T07:05:13.412 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32501.log.gz 2024-05-27T07:05:13.413 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37060.log 2024-05-27T07:05:13.413 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38116.log.gz 2024-05-27T07:05:13.414 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33134.log 2024-05-27T07:05:13.414 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42797.log.gz 2024-05-27T07:05:13.415 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35558.log 2024-05-27T07:05:13.415 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37060.log: /var/log/ceph/ceph-client.admin.33134.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37060.log.gz 2024-05-27T07:05:13.416 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33134.log.gz 2024-05-27T07:05:13.416 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34522.log 2024-05-27T07:05:13.417 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-05-27T07:05:13.418 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35558.log.gz 2024-05-27T07:05:13.418 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-05-27T07:05:13.418 INFO:teuthology.orchestra.run.smithi081.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz 2024-05-27T07:05:13.419 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28051.log 2024-05-27T07:05:13.420 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38195.log 2024-05-27T07:05:13.420 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-05-27T07:05:13.422 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42196.log 2024-05-27T07:05:13.422 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-05-27T07:05:13.422 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-05-27T07:05:13.422 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38195.log.gz 2024-05-27T07:05:13.423 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28123.log 2024-05-27T07:05:13.423 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28195.log 2024-05-27T07:05:13.423 INFO:teuthology.orchestra.run.smithi106.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28051.log.gz 2024-05-27T07:05:13.423 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28123.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-27T07:05:13.423 INFO:teuthology.orchestra.run.smithi106.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28123.log.gz 2024-05-27T07:05:13.424 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28195.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-27T07:05:13.424 INFO:teuthology.orchestra.run.smithi106.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28195.log.gz 2024-05-27T07:05:13.424 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28219.log 2024-05-27T07:05:13.426 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28147.log 2024-05-27T07:05:13.426 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28219.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-27T07:05:13.427 INFO:teuthology.orchestra.run.smithi106.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28219.log.gz 2024-05-27T07:05:13.427 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28171.log 2024-05-27T07:05:13.427 INFO:teuthology.orchestra.run.smithi106.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28147.log.gz 2024-05-27T07:05:13.427 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-27T07:05:13.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38960.log 2024-05-27T07:05:13.428 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42196.log.gz 2024-05-27T07:05:13.428 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33926.log 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37034.log 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38960.log.gz 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33926.log.gz 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37034.log.gz 2024-05-27T07:05:13.429 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-27T07:05:13.430 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-05-27T07:05:13.432 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28171.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-05-27T07:05:13.432 INFO:teuthology.orchestra.run.smithi106.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28171.log.gz 2024-05-27T07:05:13.432 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-27T07:05:13.432 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-27T07:05:13.432 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-27T07:05:13.433 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37086.log 2024-05-27T07:05:13.433 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38335.log 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.37086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37086.log.gz 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39015.log 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38335.log.gz 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34969.log 2024-05-27T07:05:13.434 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39266.log 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39015.log.gz 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40563.log 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34969.log.gz 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32917.log 2024-05-27T07:05:13.437 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39266.log.gz 2024-05-27T07:05:13.438 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph.audit.log: 93.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-27T07:05:13.438 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-27T07:05:13.438 INFO:teuthology.orchestra.run.smithi106.stderr: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-27T07:05:13.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40563.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-27T07:05:13.439 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40563.log.gz 2024-05-27T07:05:13.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38365.log 2024-05-27T07:05:13.439 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32917.log.gz 2024-05-27T07:05:13.439 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.23123.log 2024-05-27T07:05:13.440 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28099.log 2024-05-27T07:05:13.440 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-05-27T07:05:13.440 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38365.log.gz 2024-05-27T07:05:13.441 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.tmp-client.admin.23123.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.23123.log.gz 2024-05-27T07:05:13.442 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-27T07:05:13.442 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28099.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-27T07:05:13.442 INFO:teuthology.orchestra.run.smithi106.stderr: 94.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-27T07:05:13.442 INFO:teuthology.orchestra.run.smithi106.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28099.log.gz 2024-05-27T07:05:13.443 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39934.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-05-27T07:05:13.443 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39594.log 2024-05-27T07:05:13.444 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-05-27T07:05:13.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-05-27T07:05:13.446 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gz 2024-05-27T07:05:13.446 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41013.log 2024-05-27T07:05:13.447 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-05-27T07:05:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33250.log 2024-05-27T07:05:13.448 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-27T07:05:13.448 INFO:teuthology.orchestra.run.smithi081.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-05-27T07:05:13.449 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42980.log 2024-05-27T07:05:13.449 INFO:teuthology.orchestra.run.smithi081.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41013.log.gz 2024-05-27T07:05:13.450 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39344.log 2024-05-27T07:05:13.450 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33250.log.gz 2024-05-27T07:05:13.451 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37335.log 2024-05-27T07:05:13.452 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28075.log 2024-05-27T07:05:13.452 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39344.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39344.log.gz 2024-05-27T07:05:13.453 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32900.log 2024-05-27T07:05:13.454 INFO:teuthology.orchestra.run.smithi081.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.42980.log.gz 2024-05-27T07:05:13.454 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37335.log.gz 2024-05-27T07:05:13.455 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ceph-client.admin.28075.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28075.log.gz 2024-05-27T07:05:13.456 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-05-27T07:05:13.456 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42562.log 2024-05-27T07:05:13.456 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32900.log.gz 2024-05-27T07:05:13.457 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-05-27T07:05:13.457 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-05-27T07:05:13.458 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38090.log 2024-05-27T07:05:13.458 INFO:teuthology.orchestra.run.smithi081.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42562.log.gz 2024-05-27T07:05:13.459 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42482.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-05-27T07:05:13.460 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38090.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38090.log.gz 2024-05-27T07:05:13.461 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38749.log 2024-05-27T07:05:13.461 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34436.log 2024-05-27T07:05:13.463 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2024-05-27T07:05:13.463 INFO:teuthology.orchestra.run.smithi081.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38749.log.gz 2024-05-27T07:05:13.464 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-05-27T07:05:13.464 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34436.log.gz 2024-05-27T07:05:13.465 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39620.log 2024-05-27T07:05:13.465 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2024-05-27T07:05:13.467 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41566.log 2024-05-27T07:05:13.467 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-05-27T07:05:13.468 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-05-27T07:05:13.468 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39620.log.gz 2024-05-27T07:05:13.469 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37191.log 2024-05-27T07:05:13.469 INFO:teuthology.orchestra.run.smithi081.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41566.log.gz 2024-05-27T07:05:13.470 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-05-27T07:05:13.471 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-05-27T07:05:13.471 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40249.log 2024-05-27T07:05:13.471 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37191.log.gz 2024-05-27T07:05:13.472 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32346.log 2024-05-27T07:05:13.473 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-05-27T07:05:13.473 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38418.log 2024-05-27T07:05:13.474 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40249.log.gz 2024-05-27T07:05:13.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31809.log 2024-05-27T07:05:13.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32346.log.gz 2024-05-27T07:05:13.476 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39188.log 2024-05-27T07:05:13.476 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38418.log.gz 2024-05-27T07:05:13.477 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-05-27T07:05:13.477 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31809.log.gz 2024-05-27T07:05:13.478 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39802.log 2024-05-27T07:05:13.478 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39188.log.gz 2024-05-27T07:05:13.480 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42882.log 2024-05-27T07:05:13.480 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-05-27T07:05:13.481 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39882.log 2024-05-27T07:05:13.481 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39802.log.gz 2024-05-27T07:05:13.482 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42882.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42882.log.gz 2024-05-27T07:05:13.482 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2024-05-27T07:05:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39882.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39882.log.gz 2024-05-27T07:05:13.483 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42170.log 2024-05-27T07:05:13.484 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-05-27T07:05:13.485 INFO:teuthology.orchestra.run.smithi081.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2024-05-27T07:05:13.486 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-05-27T07:05:13.486 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42170.log.gz 2024-05-27T07:05:13.487 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33213.log 2024-05-27T07:05:13.487 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz 2024-05-27T07:05:13.488 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-05-27T07:05:13.488 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-05-27T07:05:13.489 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-05-27T07:05:13.489 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33213.log.gz 2024-05-27T07:05:13.491 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40537.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-05-27T07:05:13.491 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34462.log 2024-05-27T07:05:13.492 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz 2024-05-27T07:05:13.492 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2024-05-27T07:05:13.493 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32079.log 2024-05-27T07:05:13.494 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34462.log.gz 2024-05-27T07:05:13.495 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37227.log 2024-05-27T07:05:13.495 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2024-05-27T07:05:13.496 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32079.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32079.log.gz 2024-05-27T07:05:13.496 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37253.log 2024-05-27T07:05:13.497 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37227.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37227.log.gz 2024-05-27T07:05:13.497 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39724.log 2024-05-27T07:05:13.499 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41644.log 2024-05-27T07:05:13.499 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37253.log.gz 2024-05-27T07:05:13.499 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39724.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39724.log.gz 2024-05-27T07:05:13.501 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41644.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41644.log.gz 2024-05-27T07:05:13.501 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 2024-05-27T07:05:13.502 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33276.log 2024-05-27T07:05:13.503 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-05-27T07:05:13.504 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-05-27T07:05:13.504 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33276.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33276.log.gz 2024-05-27T07:05:13.506 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41801.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-05-27T07:05:13.506 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42326.log 2024-05-27T07:05:13.507 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38142.log 2024-05-27T07:05:13.508 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35003.log 2024-05-27T07:05:13.508 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42326.log.gz 2024-05-27T07:05:13.509 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38142.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38142.log.gz 2024-05-27T07:05:13.511 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35003.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35003.log.gz 2024-05-27T07:05:13.511 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-05-27T07:05:13.512 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-05-27T07:05:13.513 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38828.log 2024-05-27T07:05:13.513 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-05-27T07:05:13.516 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41434.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-05-27T07:05:13.516 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38828.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38828.log.gz 2024-05-27T07:05:13.517 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-05-27T07:05:13.517 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-27T07:05:13.518 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40909.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-05-27T07:05:13.518 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-05-27T07:05:13.520 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-05-27T07:05:13.521 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39292.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-05-27T07:05:13.522 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41039.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-05-27T07:05:13.522 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38169.log 2024-05-27T07:05:13.523 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-05-27T07:05:13.525 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38169.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38169.log.gz 2024-05-27T07:05:13.526 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42430.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-05-27T07:05:13.526 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-05-27T07:05:13.527 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-05-27T07:05:13.529 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41329.log: 91.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-27T07:05:13.529 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-05-27T07:05:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40798.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-05-27T07:05:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41959.log 2024-05-27T07:05:13.531 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32924.log 2024-05-27T07:05:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34237.log 2024-05-27T07:05:13.533 INFO:teuthology.orchestra.run.smithi081.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41959.log.gz 2024-05-27T07:05:13.534 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32924.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32924.log.gz 2024-05-27T07:05:13.535 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34237.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34237.log.gz 2024-05-27T07:05:13.535 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2024-05-27T07:05:13.536 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35490.log 2024-05-27T07:05:13.538 INFO:teuthology.orchestra.run.smithi106.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-27T07:05:13.538 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-05-27T07:05:13.538 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz 2024-05-27T07:05:13.539 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2024-05-27T07:05:13.539 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35490.log.gz 2024-05-27T07:05:13.540 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42404.log 2024-05-27T07:05:13.540 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-05-27T07:05:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41670.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2024-05-27T07:05:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40223.log 2024-05-27T07:05:13.542 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42404.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42404.log.gz 2024-05-27T07:05:13.543 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33614.log 2024-05-27T07:05:13.544 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-05-27T07:05:13.545 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40223.log.gz 2024-05-27T07:05:13.546 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33614.log: /var/log/ceph/ceph-client.admin.33160.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.33614.log.gz 2024-05-27T07:05:13.550 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34328.log 2024-05-27T07:05:13.550 INFO:teuthology.orchestra.run.smithi081.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-05-27T07:05:13.551 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2024-05-27T07:05:13.551 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-05-27T07:05:13.551 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34328.log.gz 2024-05-27T07:05:13.551 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41486.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-27T07:05:13.552 INFO:teuthology.orchestra.run.smithi081.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2024-05-27T07:05:13.552 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37447.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-05-27T07:05:13.552 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-05-27T07:05:13.552 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35276.log 2024-05-27T07:05:13.553 INFO:teuthology.orchestra.run.smithi106.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-05-27T07:05:13.554 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33187.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-05-27T07:05:13.554 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42352.log 2024-05-27T07:05:13.555 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35276.log.gz 2024-05-27T07:05:13.555 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-05-27T07:05:13.558 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42352.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42352.log.gz 2024-05-27T07:05:13.558 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-05-27T07:05:13.558 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-05-27T07:05:13.559 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42614.log 2024-05-27T07:05:13.560 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40457.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-05-27T07:05:13.561 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42142.log 2024-05-27T07:05:13.562 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42614.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42614.log.gz 2024-05-27T07:05:13.563 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42142.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42142.log.gz 2024-05-27T07:05:13.564 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37395.log 2024-05-27T07:05:13.565 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39828.log 2024-05-27T07:05:13.566 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37395.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37395.log.gz 2024-05-27T07:05:13.567 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39828.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39828.log.gz 2024-05-27T07:05:13.567 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2024-05-27T07:05:13.569 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-05-27T07:05:13.570 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42640.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2024-05-27T07:05:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39097.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-05-27T07:05:13.571 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34863.log 2024-05-27T07:05:13.573 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40091.log 2024-05-27T07:05:13.574 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34863.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34863.log.gz 2024-05-27T07:05:13.575 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40091.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40091.log.gz 2024-05-27T07:05:13.575 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34134.log 2024-05-27T07:05:13.577 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35089.log 2024-05-27T07:05:13.579 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34134.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34134.log.gz 2024-05-27T07:05:13.579 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35089.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35089.log.gz 2024-05-27T07:05:13.580 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32372.log 2024-05-27T07:05:13.581 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-05-27T07:05:13.582 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32372.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32372.log.gz 2024-05-27T07:05:13.583 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32708.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-05-27T07:05:13.584 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32503.log 2024-05-27T07:05:13.585 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38880.log 2024-05-27T07:05:13.587 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32503.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32503.log.gz 2024-05-27T07:05:13.587 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38880.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38880.log.gz 2024-05-27T07:05:13.588 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-05-27T07:05:13.589 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42456.log 2024-05-27T07:05:13.591 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40961.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-05-27T07:05:13.592 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42456.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42456.log.gz 2024-05-27T07:05:13.592 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34410.log 2024-05-27T07:05:13.593 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42852.log 2024-05-27T07:05:13.595 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34410.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34410.log.gz 2024-05-27T07:05:13.596 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42852.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42852.log.gz 2024-05-27T07:05:13.596 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-05-27T07:05:13.597 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38697.log 2024-05-27T07:05:13.599 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37616.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-05-27T07:05:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38697.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-05-27T07:05:13.600 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-05-27T07:05:13.601 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32424.log 2024-05-27T07:05:13.603 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41514.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz 2024-05-27T07:05:13.604 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32424.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32424.log.gz 2024-05-27T07:05:13.604 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-05-27T07:05:13.605 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-05-27T07:05:13.607 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39908.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-05-27T07:05:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39646.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-05-27T07:05:13.608 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2024-05-27T07:05:13.610 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35206.log 2024-05-27T07:05:13.611 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41933.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2024-05-27T07:05:13.612 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35206.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35206.log.gz 2024-05-27T07:05:13.613 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-05-27T07:05:13.614 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38309.log 2024-05-27T07:05:13.616 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32242.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-05-27T07:05:13.616 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38309.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38309.log.gz 2024-05-27T07:05:13.617 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32450.log 2024-05-27T07:05:13.618 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33303.log 2024-05-27T07:05:13.620 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32450.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32450.log.gz 2024-05-27T07:05:13.620 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33303.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33303.log.gz 2024-05-27T07:05:13.621 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-05-27T07:05:13.622 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-05-27T07:05:13.624 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40719.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-05-27T07:05:13.624 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34358.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz 2024-05-27T07:05:13.625 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32916.log 2024-05-27T07:05:13.626 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42378.log 2024-05-27T07:05:13.628 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32916.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32916.log.gz 2024-05-27T07:05:13.628 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42378.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42378.log.gz 2024-05-27T07:05:13.629 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42771.log 2024-05-27T07:05:13.630 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35833.log 2024-05-27T07:05:13.632 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42771.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42771.log.gz 2024-05-27T07:05:13.632 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35833.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35833.log.gz 2024-05-27T07:05:13.633 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32692.log 2024-05-27T07:05:13.634 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-27T07:05:13.636 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32692.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32692.log.gz 2024-05-27T07:05:13.638 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42939.log 2024-05-27T07:05:13.641 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-05-27T07:05:13.642 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32112.log 2024-05-27T07:05:13.642 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-27T07:05:13.643 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-05-27T07:05:13.644 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32112.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-05-27T07:05:13.645 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5%gzip 95.8% -- replaced with /var/log/ceph/ceph-client.admin.42939.log.gz 2024-05-27T07:05:13.645 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2024-05-27T07:05:13.645 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32112.log.gz 2024-05-27T07:05:13.647 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37309.log: 88.6%/var/log/ceph/ceph-client.admin.40746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39776.log 2024-05-27T07:05:13.648 INFO:teuthology.orchestra.run.smithi081.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2024-05-27T07:05:13.650 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39776.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-05-27T07:05:13.650 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-05-27T07:05:13.650 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37279.log 2024-05-27T07:05:13.651 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42090.log 2024-05-27T07:05:13.653 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37279.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37279.log.gz 2024-05-27T07:05:13.654 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-05-27T07:05:13.655 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42090.log.gz 2024-05-27T07:05:13.655 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-05-27T07:05:13.657 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40772.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-05-27T07:05:13.658 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39406.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-05-27T07:05:13.658 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33719.log 2024-05-27T07:05:13.659 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35180.log 2024-05-27T07:05:13.661 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33719.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33719.log.gz 2024-05-27T07:05:13.662 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35180.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35180.log.gz 2024-05-27T07:05:13.662 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35411.log 2024-05-27T07:05:13.663 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2024-05-27T07:05:13.665 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35411.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35411.log.gz 2024-05-27T07:05:13.666 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34657.log 2024-05-27T07:05:13.666 INFO:teuthology.orchestra.run.smithi081.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2024-05-27T07:05:13.667 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-05-27T07:05:13.669 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34657.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34657.log.gz 2024-05-27T07:05:13.669 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41775.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-05-27T07:05:13.670 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-05-27T07:05:13.671 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-05-27T07:05:13.673 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38281.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2024-05-27T07:05:13.674 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2024-05-27T07:05:13.674 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-05-27T07:05:13.675 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39542.log 2024-05-27T07:05:13.677 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41749.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-05-27T07:05:13.677 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39542.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39542.log.gz 2024-05-27T07:05:13.678 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34108.log 2024-05-27T07:05:13.679 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-05-27T07:05:13.681 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34108.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34108.log.gz 2024-05-27T07:05:13.681 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.31999.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-05-27T07:05:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32696.log 2024-05-27T07:05:13.683 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32398.log 2024-05-27T07:05:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32696.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32696.log.gz 2024-05-27T07:05:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32398.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32398.log.gz 2024-05-27T07:05:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-05-27T07:05:13.687 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32707.log 2024-05-27T07:05:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40169.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-05-27T07:05:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32707.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32707.log.gz 2024-05-27T07:05:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32928.log 2024-05-27T07:05:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-05-27T07:05:13.693 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32928.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32928.log.gz 2024-05-27T07:05:13.694 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-05-27T07:05:13.694 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-05-27T07:05:13.695 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-05-27T07:05:13.696 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33438.log 2024-05-27T07:05:13.697 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39986.log: /var/log/ceph/ceph-client.admin.40667.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-05-27T07:05:13.698 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-05-27T07:05:13.700 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33438.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33438.log.gz 2024-05-27T07:05:13.700 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34030.log 2024-05-27T07:05:13.701 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34004.log 2024-05-27T07:05:13.703 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34030.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34030.log.gz 2024-05-27T07:05:13.704 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34004.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34004.log.gz 2024-05-27T07:05:13.704 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-05-27T07:05:13.705 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-05-27T07:05:13.707 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32500.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz 2024-05-27T07:05:13.708 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40829.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-05-27T07:05:13.708 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32921.log 2024-05-27T07:05:13.709 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-05-27T07:05:13.711 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32921.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32921.log.gz 2024-05-27T07:05:13.712 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40405.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-05-27T07:05:13.712 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33664.log 2024-05-27T07:05:13.713 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-05-27T07:05:13.716 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33664.log: /var/log/ceph/ceph-client.admin.35594.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.33664.log.gz 2024-05-27T07:05:13.716 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42666.log 2024-05-27T07:05:13.716 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-05-27T07:05:13.717 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33383.log 2024-05-27T07:05:13.719 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42666.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42666.log.gz 2024-05-27T07:05:13.719 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33383.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33383.log.gz 2024-05-27T07:05:13.720 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39488.log 2024-05-27T07:05:13.721 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32923.log 2024-05-27T07:05:13.723 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39488.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39488.log.gz 2024-05-27T07:05:13.723 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32923.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32923.log.gz 2024-05-27T07:05:13.724 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39433.log 2024-05-27T07:05:13.725 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34384.log 2024-05-27T07:05:13.727 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39433.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39433.log.gz 2024-05-27T07:05:13.727 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34384.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34384.log.gz 2024-05-27T07:05:13.728 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-05-27T07:05:13.729 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32190.log 2024-05-27T07:05:13.731 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38557.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-05-27T07:05:13.731 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32190.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32190.log.gz 2024-05-27T07:05:13.732 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-05-27T07:05:13.733 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37112.log 2024-05-27T07:05:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39960.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-05-27T07:05:13.735 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37112.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37112.log.gz 2024-05-27T07:05:13.736 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35708.log 2024-05-27T07:05:13.737 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41091.log 2024-05-27T07:05:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35708.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35708.log.gz 2024-05-27T07:05:13.739 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41091.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41091.log.gz 2024-05-27T07:05:13.740 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27930.log 2024-05-27T07:05:13.741 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39043.log 2024-05-27T07:05:13.743 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.27930.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27930.log.gz 2024-05-27T07:05:13.743 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39043.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39043.log.gz 2024-05-27T07:05:13.744 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-05-27T07:05:13.745 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-27T07:05:13.747 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34210.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-05-27T07:05:13.747 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-27T07:05:13.749 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-27T07:05:13.750 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32497.log 2024-05-27T07:05:13.753 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32497.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32497.log.gz 2024-05-27T07:05:13.753 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2024-05-27T07:05:13.756 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40883.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2024-05-27T07:05:13.756 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32320.log 2024-05-27T07:05:13.759 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-05-27T07:05:13.760 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32320.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32320.log.gz 2024-05-27T07:05:13.760 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42116.log 2024-05-27T07:05:13.760 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-05-27T07:05:13.762 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42116.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-27T07:05:13.763 INFO:teuthology.orchestra.run.smithi106.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-27T07:05:13.763 INFO:teuthology.orchestra.run.smithi081.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42116.log.gz 2024-05-27T07:05:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41881.log 2024-05-27T07:05:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.41881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33978.log 2024-05-27T07:05:13.775 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41881.log.gz 2024-05-27T07:05:13.775 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34769.log 2024-05-27T07:05:13.775 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34082.log 2024-05-27T07:05:13.776 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33978.log.gz 2024-05-27T07:05:13.782 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34769.log: /var/log/ceph/ceph-client.admin.34082.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34082.log.gzgzip 2024-05-27T07:05:13.782 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35328.log 2024-05-27T07:05:13.782 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34769.log.gz 2024-05-27T07:05:13.784 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35128.log 2024-05-27T07:05:13.785 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35328.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35328.log.gz 2024-05-27T07:05:13.787 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-05-27T07:05:13.788 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35128.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35128.log.gz 2024-05-27T07:05:13.789 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-05-27T07:05:13.794 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41303.log: /var/log/ceph/ceph-client.admin.40197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33357.log 2024-05-27T07:05:13.794 INFO:teuthology.orchestra.run.smithi081.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-05-27T07:05:13.796 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-05-27T07:05:13.798 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33357.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-05-27T07:05:13.802 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33357.log.gz 2024-05-27T07:05:13.803 INFO:teuthology.orchestra.run.smithi106.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-27T07:05:13.804 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41696.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-05-27T07:05:13.806 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32294.log 2024-05-27T07:05:13.810 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33564.log 2024-05-27T07:05:13.812 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32294.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32294.log.gz 2024-05-27T07:05:13.814 INFO:teuthology.orchestra.run.smithi081.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-27T07:05:13.814 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42222.log 2024-05-27T07:05:13.816 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32695.log 2024-05-27T07:05:13.816 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42222.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.33564.log.gz 2024-05-27T07:05:13.819 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42222.log.gz 2024-05-27T07:05:13.820 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32695.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz 2024-05-27T07:05:13.821 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-05-27T07:05:13.823 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-05-27T07:05:13.824 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-05-27T07:05:13.825 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37519.log 2024-05-27T07:05:13.829 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37519.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37519.log.gz 2024-05-27T07:05:13.830 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34605.log 2024-05-27T07:05:13.834 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34605.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34605.log.gz 2024-05-27T07:05:13.834 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42588.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-05-27T07:05:13.834 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-05-27T07:05:13.837 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40857.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-05-27T07:05:13.837 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-05-27T07:05:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-05-27T07:05:13.840 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42011.log: 93.0% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-27T07:05:13.840 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz 2024-05-27T07:05:13.841 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35652.log 2024-05-27T07:05:13.843 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38988.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-05-27T07:05:13.843 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32701.log 2024-05-27T07:05:13.844 INFO:teuthology.orchestra.run.smithi081.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35652.log.gz 2024-05-27T07:05:13.844 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-05-27T07:05:13.846 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32701.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32701.log.gz 2024-05-27T07:05:13.846 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-05-27T07:05:13.847 INFO:teuthology.orchestra.run.smithi081.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-05-27T07:05:13.847 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-05-27T07:05:13.849 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38854.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-05-27T07:05:13.850 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40589.log 2024-05-27T07:05:13.850 INFO:teuthology.orchestra.run.smithi081.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-05-27T07:05:13.851 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-05-27T07:05:13.852 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40589.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40589.log.gz 2024-05-27T07:05:13.853 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-05-27T07:05:13.854 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-05-27T07:05:13.854 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32138.log 2024-05-27T07:05:13.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41827.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-05-27T07:05:13.856 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2024-05-27T07:05:13.856 INFO:teuthology.orchestra.run.smithi081.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32138.log.gz 2024-05-27T07:05:13.857 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38906.log 2024-05-27T07:05:13.859 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40693.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2024-05-27T07:05:13.859 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33329.log 2024-05-27T07:05:13.860 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38906.log.gz 2024-05-27T07:05:13.860 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2024-05-27T07:05:13.862 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33329.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33329.log.gz 2024-05-27T07:05:13.863 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33409.log 2024-05-27T07:05:13.863 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2024-05-27T07:05:13.863 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32486.log 2024-05-27T07:05:13.865 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33409.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33409.log.gz 2024-05-27T07:05:13.866 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-05-27T07:05:13.866 INFO:teuthology.orchestra.run.smithi081.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32486.log.gz 2024-05-27T07:05:13.867 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2024-05-27T07:05:13.868 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40143.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-05-27T07:05:13.869 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38476.log 2024-05-27T07:05:13.869 INFO:teuthology.orchestra.run.smithi081.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2024-05-27T07:05:13.871 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38476.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38476.log.gz 2024-05-27T07:05:13.872 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39698.log 2024-05-27T07:05:13.873 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-05-27T07:05:13.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39698.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39698.log.gz 2024-05-27T07:05:13.875 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38618.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-05-27T07:05:13.876 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42248.log 2024-05-27T07:05:13.877 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42274.log 2024-05-27T07:05:13.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42248.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42248.log.gz 2024-05-27T07:05:13.879 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42274.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42274.log.gz 2024-05-27T07:05:13.880 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42064.log 2024-05-27T07:05:13.881 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41907.log 2024-05-27T07:05:13.883 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42064.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42064.log.gz 2024-05-27T07:05:13.883 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41907.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41907.log.gz 2024-05-27T07:05:13.884 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37165.log 2024-05-27T07:05:13.885 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42692.log 2024-05-27T07:05:13.887 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37165.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37165.log.gz 2024-05-27T07:05:13.888 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42692.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42692.log.gz 2024-05-27T07:05:13.888 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-05-27T07:05:13.890 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32216.log 2024-05-27T07:05:13.891 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40038.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-05-27T07:05:13.892 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32216.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32216.log.gz 2024-05-27T07:05:13.892 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39568.log 2024-05-27T07:05:13.894 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-05-27T07:05:13.895 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39568.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39568.log.gz 2024-05-27T07:05:13.896 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41381.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-05-27T07:05:13.896 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38723.log 2024-05-27T07:05:13.898 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28050.log 2024-05-27T07:05:13.899 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38723.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38723.log.gz 2024-05-27T07:05:13.900 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.28050.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28050.log.gz 2024-05-27T07:05:13.901 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32164.log 2024-05-27T07:05:13.902 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-05-27T07:05:13.903 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32164.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32164.log.gz 2024-05-27T07:05:13.904 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41251.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-05-27T07:05:13.905 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-05-27T07:05:13.906 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-05-27T07:05:13.907 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40987.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-05-27T07:05:13.908 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40064.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-05-27T07:05:13.909 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32268.log 2024-05-27T07:05:13.910 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32493.log 2024-05-27T07:05:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32268.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32268.log.gz 2024-05-27T07:05:13.912 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.32493.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32493.log.gz 2024-05-27T07:05:13.913 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-05-27T07:05:13.914 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34739.log 2024-05-27T07:05:13.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38448.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz 2024-05-27T07:05:13.916 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34739.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34739.log.gz 2024-05-27T07:05:13.917 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-05-27T07:05:13.918 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-05-27T07:05:13.921 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.38670.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2024-05-27T07:05:13.921 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-05-27T07:05:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34184.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-05-27T07:05:13.928 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41855.log 2024-05-27T07:05:13.931 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33952.log 2024-05-27T07:05:13.931 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41855.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41855.log.gz 2024-05-27T07:05:13.934 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33952.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33952.log.gz 2024-05-27T07:05:13.934 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-05-27T07:05:13.937 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39380.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-05-27T07:05:13.937 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33464.log 2024-05-27T07:05:13.940 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.33464.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.33464.log.gz 2024-05-27T07:05:13.940 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2024-05-27T07:05:13.943 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2024-05-27T07:05:13.944 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41618.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2024-05-27T07:05:13.946 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40275.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-05-27T07:05:13.947 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-05-27T07:05:13.949 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40327.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-05-27T07:05:13.949 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35620.log 2024-05-27T07:05:13.952 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35620.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35620.log.gz 2024-05-27T07:05:13.954 INFO:teuthology.orchestra.run.smithi081.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-05-27T07:05:14.324 INFO:teuthology.orchestra.run.smithi081.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-27T07:05:16.268 INFO:teuthology.orchestra.run.smithi106.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-27T07:05:16.806 INFO:teuthology.orchestra.run.smithi081.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-27T07:05:17.042 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-27T07:05:17.156 INFO:teuthology.orchestra.run.smithi081.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-27T07:05:17.333 INFO:teuthology.orchestra.run.smithi081.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-27T07:05:17.364 INFO:teuthology.orchestra.run.smithi081.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-27T07:05:17.375 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-27T07:05:18.191 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-27T07:05:31.973 INFO:teuthology.orchestra.run.smithi106.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-05-27T07:05:31.975 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-05-27T07:05:31.975 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m18.589s 2024-05-27T07:05:31.975 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m31.926s 2024-05-27T07:05:31.976 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m2.494s 2024-05-27T07:05:37.442 INFO:teuthology.orchestra.run.smithi081.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.33847.log.gz 2024-05-27T07:05:37.444 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-05-27T07:05:37.444 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m24.097s 2024-05-27T07:05:37.445 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m37.155s 2024-05-27T07:05:37.445 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m3.821s 2024-05-27T07:05:37.446 INFO:tasks.ceph:Archiving logs... 2024-05-27T07:05:37.446 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi081/log 2024-05-27T07:05:37.447 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:05:41.360 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi106/log 2024-05-27T07:05:41.361 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-27T07:05:45.294 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-05-27T07:05:45.306 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-27T07:05:45.318 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/install/__init__.py", line 632, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-05-27T07:05:45.323 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-27T07:05:45.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:05:45.330 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-27T07:05:45.403 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-27T07:05:45.412 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-27T07:05:45.413 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:05:45.416 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-27T07:05:45.428 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T07:05:45.428 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2024-05-27T07:05:45.429 INFO:teuthology.orchestra.run.smithi081.stdout:+hv01.front.sepi 67.205.162.81 3 u 29 64 377 0.074 -4.445 1.016 2024-05-27T07:05:45.429 INFO:teuthology.orchestra.run.smithi081.stdout:*hv02.front.sepi 63.231.80.2 3 u 25 64 377 0.093 -4.534 0.724 2024-05-27T07:05:45.429 INFO:teuthology.orchestra.run.smithi081.stdout:+hv03.front.sepi 74.6.168.72 3 u 19 64 377 0.094 -2.719 0.944 2024-05-27T07:05:45.429 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout: remote refid st t when poll reach delay offset jitter 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================== 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout:+hv01.front.sepi 67.205.162.81 3 u 32 64 377 0.087 -6.508 0.317 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout:*hv02.front.sepi 63.231.80.2 3 u 25 64 377 0.076 -5.677 0.303 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout:+hv03.front.sepi 74.6.168.72 3 u 22 64 377 0.075 -4.685 0.319 2024-05-27T07:05:45.432 INFO:teuthology.orchestra.run.smithi106.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-05-27T07:05:45.433 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-27T07:05:45.443 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-27T07:05:45.443 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-27T07:05:45.454 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-27T07:05:45.467 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-27T07:05:45.477 INFO:teuthology.task.internal:Duration was 1933.964473 seconds 2024-05-27T07:05:45.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-27T07:05:45.489 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-27T07:05:45.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:05:45.492 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-27T07:05:45.592 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-27T07:05:45.592 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-05-27T07:05:45.592 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:05:45.613 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-05-27T07:05:45.613 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-27T07:05:45.634 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-27T07:05:45.634 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:05:45.661 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-27T07:05:45.713 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-27T07:05:45.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:05:45.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:05:45.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-27T07:05:45.839 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-27T07:05:45.839 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:05:45.859 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-27T07:05:45.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-27T07:05:45.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:05:45.908 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:05:45.922 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-05-27T07:05:45.940 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-05-27T07:05:45.968 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:05:46.001 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi081.front.sepia.ceph.com, flagging run as failed 2024-05-27T07:05:46.001 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:05:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:05:46.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-27T07:05:46.029 INFO:teuthology.task.internal:Transferring archived files... 2024-05-27T07:05:46.031 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi081 2024-05-27T07:05:46.031 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-27T07:06:00.903 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-05-27T07:06:01.342 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2024-05-27_05:52:49-fs-wip-vshankar-testing-20240521.063558-debug-testing-default-smithi/7727620/remote/smithi081/coredump/1716792958.39133.core.gz: gzip compressed data, was "1716792958.39133.core", last modified: Mon May 27 07:05:46 2024, max compression, original size modulo 2^32 365232128 2024-05-27T07:06:01.342 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-05-27T07:06:02.865 INFO:teuthology.task.internal: core looks like: /tmp/tmplb2r8mvg: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-mds', platform: 'x86_64' 2024-05-27T07:06:03.253 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2024-05-27T07:06:03.253 DEBUG:teuthology.orchestra.run.smithi081:> which 'ceph-mds -f --cluster ceph -i a' 2024-05-27T07:06:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:06:03.263 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2024-05-27T07:06:03.280 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-27T07:06:03.291 INFO:teuthology.task.internal:Not uploading archives. 2024-05-27T07:06:03.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-27T07:06:03.308 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-27T07:06:03.309 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:06:03.316 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-27T07:06:03.322 INFO:teuthology.orchestra.run.smithi081.stdout: 658125 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 27 07:05 /home/ubuntu/cephtest 2024-05-27T07:06:03.322 INFO:teuthology.orchestra.run.smithi081.stdout: 658126 4 drwxr-xr-x 4 ubuntu ubuntu 4096 May 27 06:33 /home/ubuntu/cephtest/archive 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 658128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:05 /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 655425 4 -rw-rw-rw- 1 ubuntu ubuntu 2077 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 655427 192 -rw-rw-r-- 1 ubuntu ubuntu 192819 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 655424 180 -rw-rw-rw- 1 ubuntu ubuntu 183553 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 658127 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 06:55 /home/ubuntu/cephtest/archive/coredump 2024-05-27T07:06:03.323 INFO:teuthology.orchestra.run.smithi081.stdout: 655428 132832 -rw------- 1 root root 365232128 May 27 06:55 /home/ubuntu/cephtest/archive/coredump/1716792958.39133.core 2024-05-27T07:06:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-27T07:06:03.324 INFO:teuthology.orchestra.run.smithi081.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-27T07:06:03.325 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_38f9d900c83c63799fbdbe61acc9a11b0d3554a6/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-05-27T07:06:03.339 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-27T07:06:03.349 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-27T07:06:03.366 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-05-27T07:06:03.371 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 1933.9644730091095 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi081 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=38f9d900c83c63799fbdbe61acc9a11b0d3554a6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default mon.a-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 mon.b-kernel-sha1: b9459a26b0888785d05c47deb35abc9d0354ab93 owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cad2714540fc496eb2736d3d148b6084 status: fail success: false 2024-05-27T07:06:03.372 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-27T07:06:03.379 INFO:teuthology.orchestra.run.smithi106.stdout: 655428 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 27 07:05 /home/ubuntu/cephtest 2024-05-27T07:06:03.380 INFO:teuthology.orchestra.run.smithi106.stdout: 658125 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 27 07:05 /home/ubuntu/cephtest/archive 2024-05-27T07:06:03.380 INFO:teuthology.orchestra.run.smithi106.stdout: 658128 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 27 07:05 /home/ubuntu/cephtest/archive/syslog 2024-05-27T07:06:03.380 INFO:teuthology.orchestra.run.smithi106.stdout: 658130 4 -rw-rw-rw- 1 ubuntu ubuntu 1843 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-05-27T07:06:03.380 INFO:teuthology.orchestra.run.smithi106.stdout: 658129 184 -rw-rw-r-- 1 ubuntu ubuntu 185093 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-27T07:06:03.381 INFO:teuthology.orchestra.run.smithi106.stdout: 658131 172 -rw-rw-rw- 1 ubuntu ubuntu 172617 May 27 07:05 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-05-27T07:06:03.381 INFO:teuthology.orchestra.run.smithi106.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-05-27T07:06:03.457 INFO:teuthology.run:FAIL